[01:35:16] 10Analytics, 03Community-Tech-Sprint: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3290926 (10Niharika) 05Open>03Resolved I believe this can be resolved now. This time the bot ran through all projects except Biography in 15 days. The updat... [06:54:55] 10Analytics, 10DBA, 06Labs, 10MediaWiki-Page-deletion, 10Tool-Labs-tools-Database-Queries: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3290992 (10jcrespo) > what's the relation between dbstore1001 and and analytics-s... [07:18:41] hallo! [07:19:12] so today I'd like to restart yarn/hdfs daemons for jdk upgrades and merge the code review to refactor the aqs puppet code [07:19:20] should be a no op for both [07:19:37] let me know if you have anything against it due to other things to do [07:21:41] Hi elukey - Please go forward :) [07:23:43] !log Restart pageview-druid-hourly-wf-2017-5-24-19 [07:23:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:24:12] Seems we are experiencing issues with Druid lading jobs (hadoop jobs started by Druid itself) since yesterday [07:24:47] joal: weird, it should be ok now since Druid has been rolled back [07:24:54] what kind of issues? [07:29:45] Same as after upgrade, but not on every mapper/reducer [07:30:13] elukey: example of a failed job : https://yarn.wikimedia.org/jobhistory/job/job_1492691387549_113100 [07:30:32] But we have many with some failed mapper and reducer that succed at the end [07:31:48] weird [07:32:13] so IIRC Andrew re-build 0.9 packages, removed 0.10 and restored 0.9 [07:32:50] so it wasn't a clean rollback stricly speaking, meaning that there might be the possibility that something is a bit different from the "old" 0.9 version [07:32:54] (shouldn't be but..) [07:33:48] elukey: I suspect something like that yes [09:12:18] 10Analytics, 10DBA, 06Labs, 10MediaWiki-Page-deletion, 10Tool-Labs-tools-Database-Queries: Database replication issues with deleted pages (affecting Tool Labs and Analytics Store) - https://phabricator.wikimedia.org/T166194#3291248 (10jcrespo) To clarify my previous statement, analytics hosts are conside... [09:35:26] joal: started the yarn nodemanager restarts fyi [09:35:42] K elukey, thanks for pinging [09:35:46] (two at the time, 60 seconds of pause between each batch) [09:35:54] will do hdfs datanodes after that [09:36:01] aaand 1030 is back! [09:36:05] so I can reimage it [09:36:14] elukey: let me know when finished, I'll double check oozie after [09:36:46] ack [09:38:14] (03PS1) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 [09:46:05] (03PS2) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 [09:52:26] elukey: about druid loading issues, would if possible that we forgot to rollback HDFS extension somewhere? [09:55:45] joal: mmmm I can check now [10:00:55] joal: ls -l /usr/share/druid/extensions/druid-hdfs-storage-cdh --color looks good on druid1001 [10:01:12] (nodemanagers restarted, proceeding with datanodes) [10:01:44] elukey: on druid1003, there are both 0.10 and 0.9 [10:02:04] gneee [10:02:04] maybe it might be the reason ? [10:02:18] Have not checked 1002 [10:02:22] I was checking 1002 [10:03:19] joal: what do you mean with "there are both etc.." ? [10:03:58] well, in the path you provided, there are 2 jars for hdfs extension: one named after version 0.9.0, the other 0.10.0 [10:04:38] ahhhhh in the paths! [10:05:01] elukey: I assume the thing uses a folder in teh classpath, and loads everything [10:05:11] I can try re-running the script to generate the extension [10:05:42] elukey: I'd just go with deleting the 0.10 version [10:06:41] let me check what /usr/local/bin/druid-hdfs-storage-cdh-link does [10:10:28] so the 0.10 is definitely a broken symlink [10:10:30] removing it [10:11:39] !log removed /usr/share/druid/extensions/druid-hdfs-storage-cdh/druid-hdfs-storage-0.10.0.jar from all druid nodes [10:11:40] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:14:17] joal: shall we restart each daemon one at the time? [10:14:24] like we do for rolling restarts [10:14:50] elukey: I don't know - let it run this way for the moment, I have indexation tasks running - I'll check if they fail [10:16:30] mmmm [10:16:43] in the meantime, all datanodes restarted [10:17:08] (03PS1) 10Mforns: Add script to puge old mediawiki data snapshots [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355601 (https://phabricator.wikimedia.org/T162034) [10:20:03] awesome [10:20:28] elukey: No oozie complains - You my ops people are really good at making our stuff reliable :) [10:20:45] * joal is glad to have such good coworkers :) [10:21:53] elukey: your cleaning has worked - No eratic failing anymore - Thanks mate [10:22:34] joal: sorry it is still in progress I misread, a couple of nodes still to go [10:22:44] time for oozie to complain :D [10:22:50] np elukey - Was talking about druid loading [10:22:54] :) [10:23:11] * joal warms up for an oozie fight [10:27:25] all right this time is done! [10:27:34] will need to do master nodes but later on [11:05:24] * elukey lunch! [11:38:02] 06Analytics-Kanban, 13Patch-For-Review: Update druid to latest release - https://phabricator.wikimedia.org/T164008#3291601 (10JAllemandou) This was >>! In T164008#3290363, @Ottomata wrote: > Oof, Hadoop druid loading jobs are still failing, even after rolling back: Problem found: There were broken links to ve... [11:42:56] joal: --^ stat1003 ? [11:43:06] druid1003 - correcting [11:43:08] I removed the symlink in all the druid nodes btw [11:43:41] corrected [11:55:04] thanks! [11:55:49] BTW elukey, we have webrequests loaded regularly in druid (just for fun) [11:57:09] niceeeeeee [11:57:19] sampling? [11:57:43] 1/128 [12:02:27] joal: are you around if I merge https://gerrit.wikimedia.org/r/#/c/354107 ? [12:02:35] my plan is to do it only on aqs1004 [12:02:42] restart one of the instances [12:02:51] and check that nothing explodes [12:03:13] it shouldn't but better safe than sorry [12:03:44] elukey: I'm here, but I can't see how I'd be helpful :) [12:04:20] two more eyes are good :) [12:04:27] sounds good :) [12:04:40] elukey: let me know if you want me to look at something special [12:05:05] will do [12:10:29] Will wait for Filippo for the final signoff [12:10:38] okey [12:11:01] elukey: something else to have fun with: https://pivot.wikimedia.org/#netflow [12:11:06] ;) [12:32:34] elukey: put it in puppet, mayyybe another repo, but don't make a deb! [12:32:45] hiiii:) [12:39:29] joal: it was just hte broken links?!?!?! [12:39:36] ottomata: looks son [12:39:43] i saw those, but kind of ignored them [12:39:46] i should have just removed them [12:39:55] it's ok :) [12:39:57] how could the broken links cause a unsupported version problem? [12:40:09] they didn't point to anything! [12:40:11] maybe it was not broken everywhere? [12:40:16] IO have no idea [12:40:18] i checked all 3 druids [12:40:19] hm [12:40:21] so weird [12:40:36] 06Analytics-Kanban, 13Patch-For-Review: Update druid to latest release - https://phabricator.wikimedia.org/T164008#3291674 (10Ottomata) Totalllly weeiiiiiird! Thanks for finding that! [12:40:39] joal: ok, so loading jobs are cool now? [12:40:44] ottomata: yes sir ! [12:40:59] great [12:42:00] oh i think tranquility is workin gnow too! [12:42:10] yes! [12:42:11] receivedCount=260768, sentCount=260768 [12:42:26] ottomata: I diddn't restart banners, but started the one for paravoid [12:42:36] ottomata: [12:42:44] i just restarted banners [12:42:44] ottomata: https://pivot.wikimedia.org/#netflow [12:42:45] and now its working [12:42:48] great [12:42:52] h cooool [12:43:18] haha, cool, dunno what these numbers are [12:43:19] but awesome [12:43:23] joal: where is that tranquility running? [12:43:34] ottomata: screen on druid1003 [12:45:45] k great [12:46:56] ottomata: hi! What do you mean don't make a deb? [12:47:37] for the eventlogging purge [12:47:46] just saw some of your convo in mw-sec [12:48:22] yeah, but I don't see a good alternative [12:48:30] scap? git::clone? [12:48:38] just puppet [12:48:52] i'd really do just puppet, but it sounded like everyone else said do it elsewher [12:48:56] well I need to add tests, and all in one file seems not the right target for puppet [12:49:00] i don't see anything wrong with having it all in just one file [12:49:12] it is horrible :D [12:49:16] nawww its not [12:49:30] this time you can't change my mind, it is horrible :D [12:49:31] but, i think more people agree with you than me [12:49:33] haha [12:49:34] yeah [12:49:54] maybe i haven't seen the end result, but i think its fine for one off scripts to have tests inside of them, kind of nice even with python [12:49:59] since usually the entry point is the __main__ thing [12:49:59] I created https://gerrit.wikimedia.org/r/#/c/355604/1 for the moment, just as proposal [12:50:26] right, but no test yet ya? [12:50:53] i guess i just think that having all the overhead of a module and a repo and a deployment for something that is a one off is a lot [12:50:53] no no still haven't reached that point [12:50:54] but anyway [12:50:58] i'm not trying to convince you not to do repo [12:51:00] just not to do deb :) [12:51:17] yes the deb is a bit overkill, I agree [12:51:23] I thought it was the only alternative [12:51:53] i think probably a puppet git::clone is the easiest [12:52:14] ottomata: I'll try to see how long the tests are, in case they are tiny I'll just use a single horrible file [12:52:20] in the meantime I'll keep the repo [12:52:30] aye [12:52:31] worst that happens is that I'll delete it [12:52:54] :) I'll leave that up to you, i think you like the repo :) [12:53:28] ottomata: you know that I always follow your advices, so I'll probably not use it in the end [12:54:01] haha [12:54:03] noooooo [12:54:09] cmooon you can do it if you want to! [12:54:09] haha [12:54:10] :P [12:54:13] others in ops liked your repo idea [12:54:15] hahahaa [12:54:16] don't just listen to me! [12:54:30] I am clouded by your judgement! Going to write it in the peer review [12:54:35] ahhahaha [12:54:44] joal: going to merge the aqs patch! [12:54:45] https://gerrit.wikimedia.org/r/#/c/354107/ [12:54:59] okey [12:55:06] hhahah [12:58:09] (03CR) 10Ottomata: [C: 031] "Only did a quick review, but the general idea and layout looks good to me!" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/350219 (https://phabricator.wikimedia.org/T157582) (owner: 10Mforns) [12:58:38] applied the change to AQS [12:58:41] err aqs1004 [12:58:57] going to wait a bit to see if ferm rules are ok or blocking [12:59:04] and then I'll restart one instance [13:00:00] (03PS2) 10Ottomata: Fix broken select_missing_sequence_run_query [analytics/refinery] - 10https://gerrit.wikimedia.org/r/351666 [13:00:06] (03CR) 10Ottomata: [V: 032 C: 032] Fix broken select_missing_sequence_run_query (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/351666 (owner: 10Ottomata) [13:00:55] (03PS3) 10Ottomata: Add README.mediawiki-tables-sqoop-orm [analytics/refinery] - 10https://gerrit.wikimedia.org/r/351667 (https://phabricator.wikimedia.org/T143119) [13:01:14] (03CR) 10Ottomata: [V: 032] Add README.mediawiki-tables-sqoop-orm [analytics/refinery] - 10https://gerrit.wikimedia.org/r/351667 (https://phabricator.wikimedia.org/T143119) (owner: 10Ottomata) [13:02:23] (03Abandoned) 10Ottomata: Updates to pivot 0.11.33 from Imply distribution 2.1.1 [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/351450 (https://phabricator.wikimedia.org/T164007) (owner: 10Ottomata) [13:03:26] (03Abandoned) 10Ottomata: Remove analytics1027 from scap targets, it will be decomissioned [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/344491 (https://phabricator.wikimedia.org/T159527) (owner: 10Ottomata) [13:04:47] 10Analytics, 06Operations, 10Traffic, 13Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3291750 (10Ottomata) Just nocied this ticket again! Can/should we move forward with this? [13:10:18] joal: everything seems good, rolling out the change to all the aqs nodes [13:10:25] k elukey [13:10:35] big chunk of puppet tech debt removed from Giuseppe :) [13:10:56] 10Analytics, 06Operations, 10Traffic, 13Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#3291758 (10BBlack) From my perspective, where we last stalled out is waiting for Analytics to say it's ok to merge https://gerrit.wikimedia.org/r/#/c/2... [13:14:05] Taking a break until standup guys [13:32:27] 10Analytics, 10EventBus, 10Wikimedia-Logstash, 05MW-1.29-release (WMF-deploy-2016-11-01_(1.29.0-wmf.1)), and 2 others: Type collisions in log events causing indexing failures in ELK Elasticsearch - https://phabricator.wikimedia.org/T150106#3291814 (10Krinkle) [13:34:31] 10Analytics, 10EventBus, 10Wikimedia-Logstash, 05MW-1.29-release-notes, 13Patch-For-Review: Type collisions in log events causing indexing failures in ELK Elasticsearch - https://phabricator.wikimedia.org/T150106#3291821 (10Krinkle) [14:02:24] 10Analytics, 10MediaWiki-API, 06Wikipedia-Android-App-Backlog, 06Wikipedia-iOS-App-Backlog: Add page_id and namespace to X-Analytics header in App / api requests - https://phabricator.wikimedia.org/T92875#3291933 (10Krinkle) [14:04:43] 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#3291938 (10Krinkle) [14:18:31] hey a-team, I'm still not feeling awesome but I'll try to work today. I may miss standup though because I'm at the baby doctor [14:19:17] (03CR) 10Mforns: ">" (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/350219 (https://phabricator.wikimedia.org/T157582) (owner: 10Mforns) [14:19:45] (03Abandoned) 10Mforns: Add monthly sanitized job for banner activity [analytics/refinery] - 10https://gerrit.wikimedia.org/r/350219 (https://phabricator.wikimedia.org/T157582) (owner: 10Mforns) [14:37:08] vagrant@debian-jessie:~/analytics-eventlogging-maintenance$ python setup.py test [14:37:11] running test [14:37:12] so nice [14:37:21] :P [14:45:22] elukey, \\o// [14:45:50] mforns: did you see the new patch? The repo is cleaner in my opinion [14:46:05] elukey, yes I saw :] [14:46:14] as Andrew was saying, maybe we could deploy it via scap? [14:47:08] 10Analytics, 10Analytics-EventLogging, 06Community-Tech, 06Editing-Analysis, and 2 others: Record an EventLogging event every time a new mainspace page is created - https://phabricator.wikimedia.org/T150369#3292122 (10Nuria) >Another reason we would like to have this data in EventLogging is that it would b... [14:49:04] elukey, yea ok, dunno though if we're going to develop that script a lot... I mean, probably it's going to see few deployments no? [14:49:59] elukey: see script on puppet plus tests on file for performance scripts we run https://github.com/wikimedia/puppet/blob/production/modules/webperf/files/navtiming.py#L419 [14:51:12] elukey: this is the approach we use for light scripts such us this one, so having tests in file I think is pretty standard if what you are deploying is just a script and not an app [14:51:45] nuria_: well it is a bit more than a script, and tests are not so tiny as the example you showed to me [14:52:00] I mean, I can squeeze everything in there but it seems a bit forced [14:52:22] the rule in puppet is to store relatively small scripts without propert repos only [14:52:56] in any case, this is only a test to ease the deployment, in the end we'll see if everything is squeezable in one file :) [14:53:05] *development [14:54:05] (some numbers - the script alone is ~300 lines of code, the first file for testing is ~50) [14:55:38] (plus the repo is generic enough to hold other scripts, like eventlogging_sync.sh, etc..) [14:56:29] elukey: ok, up to you buy just so you know we do this for important stuff (that script is in charge of most of our performance data for mediawiki releases) [14:58:20] nuria_: sure, I'll do whatever the team thinks is best, deleting the repo (that is empty) is a matter of 10 seconds :) [14:58:53] elukey: nah, you can do as you think is best, this was just an FYI so you know it is not unheard of [15:01:03] ping mforns standduppp [15:07:50] urandom: FYI https://gerrit.wikimedia.org/r/#/c/354107/ is now live [15:08:09] 06Analytics-Kanban, 13Patch-For-Review: Refactor monthly banner oozie job to use already indexed daily data - https://phabricator.wikimedia.org/T159727#3292203 (10mforns) a:05JAllemandou>03mforns [15:27:31] ping joal [15:36:31] 10Analytics, 06Operations, 10Traffic, 13Patch-For-Review: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557#1803407 (10Nuria) 05duplicate>03Open [15:41:46] milimetric: we forgot to talk about autoconfirmed stuff [15:42:51] milimetric: see also https://phabricator.wikimedia.org/T150369 [15:43:42] elukey: cool! [15:45:12] nuria_: yeah, getting extremely accurate exact data exactly the way they want is a little tricky. EventLogging would help, but I think the problem is the queue growing, not lack of data [15:45:32] milimetric: also data accurate to 0.01% is not going to help [15:45:37] and we're spending hundreds of hours trying to figure out how to explain what's going on instead of fixing [15:45:41] exactly [15:45:47] milimetric: i bet if you compute variability of that number is pretty high [15:46:21] yeah, it does matter very much how exactly it's measured too, beyond that it's kind of noisy [15:46:36] but ultimately who cares? The community is in pain and we're arguing semantics [15:48:49] 06Analytics-Kanban, 13Patch-For-Review: Provide uniques estimate/offset breakdowns available in dumps - https://phabricator.wikimedia.org/T164597#3292371 (10Nuria) [15:49:02] 06Analytics-Kanban, 13Patch-For-Review: Load pivot pageview-hourly dataset every hour - https://phabricator.wikimedia.org/T164730#3292372 (10Nuria) [15:51:01] elukey, joal : tasking/grooming/whatever else [16:00:34] 06Analytics-Kanban: Evaluate swiv and see whether outstanding pivot bugs are fixed - https://phabricator.wikimedia.org/T166320#3292409 (10Nuria) [16:00:55] 06Analytics-Kanban: Evaluate swiv and see whether outstanding pivot bugs are fixed - https://phabricator.wikimedia.org/T166320#3292434 (10Nuria) p:05Triage>03High [16:03:16] 06Analytics-Kanban: Coordinate with research to vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#3292464 (10Nuria) [16:03:28] 06Analytics-Kanban: vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#2895122 (10Nuria) [16:05:49] 06Analytics-Kanban: upgrade druid and pivot - https://phabricator.wikimedia.org/T157977#3292470 (10Nuria) [16:05:51] 06Analytics-Kanban, 13Patch-For-Review: Update pivot to latest source - https://phabricator.wikimedia.org/T164007#3292469 (10Nuria) 05Open>03declined [16:07:14] 10Analytics, 10EventBus, 10Wikimedia-Stream, 06Services (watching), 15User-mobrovac: Bikeshed what events should be exposed in public EventStreams API - https://phabricator.wikimedia.org/T149736#3292485 (10Nuria) [16:10:22] 06Analytics-Kanban: AQS alarms need to log to analytics channel - https://phabricator.wikimedia.org/T162407#3292492 (10Nuria) 05Open>03declined After refactor of AQS puppet code this no longer applies. [16:11:55] 10Analytics: Investigate the use of local_quorum for AQS - https://phabricator.wikimedia.org/T164348#3292501 (10Nuria) [16:12:24] 10Analytics: Provide unqiues estimate/offset breakdowns in AQS - https://phabricator.wikimedia.org/T164593#3292503 (10Nuria) [16:19:11] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Update kafka.sh wrapper script for Kafka 0.10+ - https://phabricator.wikimedia.org/T166164#3292525 (10Nuria) [16:19:34] 10Analytics-Cluster, 06Analytics-Kanban, 13Patch-For-Review: Update puppet for new Kafka cluster and version - https://phabricator.wikimedia.org/T166162#3292526 (10Nuria) [16:22:30] 06Analytics-Kanban, 10DBA, 06Operations: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3292531 (10Nuria) p:05Triage>03High [16:27:06] 10Analytics: Code Review Needed: New data produced on https://analytics.wikimedia.org/datasets/ - https://phabricator.wikimedia.org/T165944#3292571 (10Nuria) @GoranSMilovanovic Please be so kind to 1) document the data you will be exposing so we can study whether it can be public 2) publish your code where we... [16:27:27] 06Analytics-Kanban: Code Review Needed: New data produced on https://analytics.wikimedia.org/datasets/ - https://phabricator.wikimedia.org/T165944#3292574 (10Nuria) [16:30:17] 10Analytics, 10Analytics-Wikistats, 10Wikimedia-Site-requests: Add li: Wikibooks to Wikistats - https://phabricator.wikimedia.org/T165634#3272286 (10Nuria) We are working on the new version of Wikistats, this site will automatically be included. [16:33:05] 10Analytics, 10DBA, 07Chinese-Sites: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3292614 (10Nuria) [16:35:54] 10Analytics, 10DBA, 07Chinese-Sites: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3292618 (10Nuria) Ping @Marostegui: do we have an ETA on when these wikis will be available on labs new db hosts? Ping @Neil_P._Quinn_WMF this data is on production snapshot (we are do... [16:36:24] 10Analytics, 10Analytics-Wikistats, 10Wikimedia-Site-requests: Add li: Wikibooks to Wikistats - https://phabricator.wikimedia.org/T165634#3272286 (10Nemo_bis) Uh, wiki created in 2010 by JeLuF. The newprojects list is older than I thought though, and this list was properly announced: https://lists.wikimedia.... [16:39:53] 10Analytics, 10DBA, 07Chinese-Sites: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3260779 (10jcrespo) > do we have an ETA on when these wikis will be available on labs new db hosts? We are thinking by the end of FQ1, according to our roadmap. [16:40:00] 10Analytics: Create small sample mediawiki-history table in MariaDB - https://phabricator.wikimedia.org/T165309#3262603 (10Nuria) We can do a one off and see if it helps [16:47:00] 10Analytics, 10DBA, 07Chinese-Sites: Data Lake edit data missing for many wikis - https://phabricator.wikimedia.org/T165233#3292656 (10Nuria) Ok, so we can plan on this data being available in September, correct? Until then we will continue taking snapshots of production and labs. [17:01:36] bearloga: sync up meeting? [17:02:33] nuria_: i think i clicked on the wrong thing. i was in the wrong hangout waiting for you :P [17:23:52] 10Analytics, 10Analytics-Wikistats, 10Wikimedia-Site-requests: Add li: Wikibooks to Wikistats - https://phabricator.wikimedia.org/T165634#3292726 (10Ooswesthoesbes) Indeed, it has not appeared on Wikistats ever since 2010. [17:25:19] bearloga: ah sorry [17:27:21] Heya, here now [17:27:29] Tell me folks, what did I miss? [17:30:25] 10Analytics, 10Analytics-Wikistats, 10Wikimedia-Site-requests: Add li: Wikibooks to Wikistats - https://phabricator.wikimedia.org/T165634#3292742 (10Nuria) Is this the site? https://analytics.wikimedia.org/dashboards/vital-signs/#projects=liwikibooks/metrics=Pageviews Pageviews should be available both dai... [18:00:34] * elukey off! [18:01:12] ottomata: just realized that the kafka hw meeting is at 10 pm for me :( [18:01:17] I am not probably going to make it [18:06:34] joal: did you ping ema about varnish cpokies? [18:06:37] *cookies [18:06:42] oooooof elukey yeah, yargh [18:06:46] so hard to coordinate meetings sometimes [18:06:48] elukey: that's ok [18:07:01] i'm going to suggest buying more ram, and we are going to talk about SSds [18:07:02] for main kafka [18:07:05] that is probably going to be it though [18:07:14] mayyybe we'll use some of that money to buy some new scb node hosts [18:07:15] dunno [18:10:58] 10Analytics: Pivot - Article Page Views - https://phabricator.wikimedia.org/T166331#3292854 (10Jseddon) [18:11:06] 10Analytics, 10Analytics-Cluster, 06Operations, 10ops-eqiad: rack/setup/install replacement to stat1005 (stat1002 replacement) - https://phabricator.wikimedia.org/T165368#3264256 (10Cmjohnson) @Ottomata It is in the rack as of today and I am getting through several orders and will do my best. I don't have... [18:11:27] 10Analytics: Pivot - Article Page Views - https://phabricator.wikimedia.org/T166331#3292868 (10Jseddon) [18:22:38] 10Analytics: Pivot - Article Page Views - https://phabricator.wikimedia.org/T166331#3292854 (10Nuria) @Jseddon: Data for pageviews per hour with page titles updated hourly exists, but note that is a tremendously huge dataset, it is only available on hadoop and query-able through command line. The pageview api w... [18:33:54] (03PS4) 10Nuria: [WIP] UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) [18:42:03] ottomata: re T118557, just to make sure: if we drop XFF header, we still will have referrer info as part of webrequest logs, but it will get collected at a different place, correct? that's a useful field for us and I want to make sure we continue receiving that information. [18:42:04] T118557: Replace Analytics XFF/client.ip data with X-Client-IP - https://phabricator.wikimedia.org/T118557 [18:43:45] lzia: referrer is totally different that xff [18:43:46] right? [18:44:09] the client_ip field which exists in the webrequest table is now computed from the X-Client-IP header which varnish is setting for us [18:44:50] 06Analytics-Kanban: Test failures in refinery master - https://phabricator.wikimedia.org/T166334#3292921 (10Nuria) [18:45:26] isn't referrer info in x_forwarded_for header, ottomata? [18:45:39] got you, ottomata. [18:46:36] lzia referer is the site/page the person came from, no? [18:46:43] xff is a list of IPs [18:46:52] that hopefully contain the client's real IP address [18:46:55] but also have other proxy IPs in it [18:47:07] hmm, I confused it, ottomata then. thanks for walking me through it. [18:47:12] yuppers :) [18:53:26] 10Analytics, 13Patch-For-Review: Update druid to latest release - https://phabricator.wikimedia.org/T164008#3292943 (10Ottomata) [19:03:57] 10Analytics, 10Analytics-Dashiki, 07Wikimedia-log-errors: Warning: JsonConfig: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T166335#3292968 (10thcipriani) [19:14:20] Gone for tonight a-team, sorry for having missed groosking [19:15:26] laters joal! [19:18:17] (03PS5) 10Nuria: [WIP] UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) [19:18:35] jajajaj [19:18:37] groskin [19:18:42] grooskin [19:18:45] that is awesome [19:26:30] (03PS6) 10Nuria: [WIP] UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) [19:50:36] 10Analytics: Are watchlists dead? - https://phabricator.wikimedia.org/T166339#3293102 (10Whatamidoing-WMF) [19:59:02] 06Analytics-Kanban: Code Review Needed: New data produced on https://analytics.wikimedia.org/datasets/ - https://phabricator.wikimedia.org/T165944#3293134 (10GoranSMilovanovic) @Nuria 1. Data: counts the usage of Wikidata items from wbc_entity_usage tables on 20 client projects; check lines 99 - 104 in the R c... [20:15:02] 06Analytics-Kanban: Code Review Needed: New data produced on https://analytics.wikimedia.org/datasets/ - https://phabricator.wikimedia.org/T165944#3293166 (10Nuria) @GoranSMilovanovic See here an example of a dataset description: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Webrequest or http... [20:16:26] 10Analytics: Are watchlists dead? - https://phabricator.wikimedia.org/T166339#3293102 (10Elitre) Trizek will find out more about this soon, no? [20:22:18] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#2995179 (10Ottomata) [20:22:21] 10Analytics, 10EventBus, 06Operations, 10hardware-requests, and 2 others: SSDs for main Kafka clusters - https://phabricator.wikimedia.org/T166341#3293176 (10Ottomata) [20:23:32] ottomata: boy, deps are hard no matter where, we have like 4 different versions of guava [20:24:55] haha [20:24:56] yay! [20:27:26] 10Analytics, 10ChangeProp, 10EventBus, 06Services (later), 15User-mobrovac: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#3293212 (10Ottomata) [20:27:28] 10Analytics, 10EventBus, 06Services (later), 15User-mobrovac: New SCB nodes - https://phabricator.wikimedia.org/T166342#3293197 (10Ottomata) [20:27:46] 10Analytics, 10EventBus, 06Services (later), 15User-mobrovac: New SCB nodes - https://phabricator.wikimedia.org/T166342#3293217 (10Ottomata) [20:33:28] 06Analytics-Kanban: Code Review Needed: New data produced on https://analytics.wikimedia.org/datasets/ - https://phabricator.wikimedia.org/T165944#3293239 (10GoranSMilovanovic) @Nuria here's another try: - the script at https://github.com/wmde/WDCM/blob/master/WDCM_Search_Clients.R (running locally on stat10... [20:36:58] 10Analytics: Are watchlists dead? - https://phabricator.wikimedia.org/T166339#3293242 (10Elitre) Also, the stats for a given page (Page Information link) may shed some light on this, see the following for the VE board at en.wp; Number of page watchers 339 Number of page watchers who visited recent edits 98 [20:39:59] 10Analytics, 10EventBus, 06Operations, 10hardware-requests, and 2 others: SSDs for main Kafka clusters - https://phabricator.wikimedia.org/T166341#3293256 (10RobH) SSDs in this timeline isn't possible, not if we want them under warranty with the system vendor . [20:40:38] 10Analytics, 10EventBus, 06Operations, 10hardware-requests, and 2 others: SSDs for main Kafka clusters - https://phabricator.wikimedia.org/T166341#3293262 (10RobH) a:03Ottomata Is this something that you want done in next years budget, or is it now invalid? Please advise. [20:41:21] 10Analytics, 10EventBus, 06Services (later), 15User-mobrovac: New SCB nodes - https://phabricator.wikimedia.org/T166342#3293266 (10RobH) So, do you want these additional servers to have SATA (like SCB has now) or SSD (like you want SCB to eventually have?) [20:45:07] 10Analytics, 10EventBus, 06Services (later), 15User-mobrovac: New SCB nodes - https://phabricator.wikimedia.org/T166342#3293277 (10RobH) I've tagged in #hardware-requests as otherwise I would miss this task (except @ottomata pinged me about it ;) I'll create a #procurement task and get a quote update on it. [20:45:34] 10Analytics, 10EventBus, 06Operations, 10hardware-requests, and 2 others: New SCB nodes - https://phabricator.wikimedia.org/T166342#3293278 (10RobH) [20:46:10] 10Analytics, 10EventBus, 06Operations, 10hardware-requests, and 2 others: New SCB nodes - https://phabricator.wikimedia.org/T166342#3293284 (10Ottomata) Thanks @RobH [20:46:12] 10Analytics: Are watchlists dead? - https://phabricator.wikimedia.org/T166339#3293285 (10Varnent) >>! In T166339#3293242, @Elitre wrote: > Also, the stats for a given page (Page Information link) may shed some light on this, see the following for the VE board at en.wp; > > Number of page watchers 339 > Number o... [20:48:25] 10Analytics, 06Research-and-Data: clientIP needs to be collected as part of the schema or ... - https://phabricator.wikimedia.org/T165678#3293295 (10leila) p:05Unbreak!>03High [20:50:46] 10Analytics: Are watchlists dead? - https://phabricator.wikimedia.org/T166339#3293300 (10Whatamidoing-WMF) I believe that "watchers who visited recent edits" is more like "on my watchlist, and my account had any (logged-in reading) activity at all". I don't believe that it shows the number of people who looked... [20:54:45] ottomata: ok, down to 1 version of guava, let's see if the rest of stuff still works [20:54:52] nice [21:14:13] ottomata: still that is on our code, i bet old guava is going to be loaded by hadoop's own deps [21:16:44] ottomata: where is hive-site on 1002? [21:33:33] ottomata: ah, no, we cannot use the version of guava we want looks like hadoop hasn't updated its guava dep in years: https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-yarn/hadoop-yarn-client/dependency-analysis.html [21:33:50] ottomata: so we need to use guava 11 [21:35:27] aye [21:35:31] nuria_: hive-site is /etc/hive/conf [21:35:58] ottomata: nvm just printed classpath and read all bugs about hadoop and guava 11, looks like it is there to stay [21:49:03] (03CR) 10Bearloga: [C: 04-1] [WIP] UDF to tag requests (035 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [22:02:51] (03CR) 10Nuria: [WIP] UDF to tag requests (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [23:05:00] nuria_: i'm making some changes to PortalTagger.java right now, btw :) [23:16:42] (03PS7) 10Bearloga: [WIP] UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [23:17:29] (03PS8) 10Bearloga: [WIP] UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria)