[05:57:45] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3314068 (10Marostegui) This BBU failed again and the policy went back to WriteThrough: ``` Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad... [07:01:34] hello! [07:55:53] Hi elukey [07:59:27] bonjour Joseph [08:00:04] What's up friend? [08:00:31] I am writing reviews [08:00:44] Arf, yes - I should do that to [08:00:45] after that I need to work on code reviews [08:00:49] ahahaha [08:00:56] a day of joy [08:01:21] Best way to start a wekk: after that, anything will be fun :) [08:14:39] 10Analytics: Browser analytics dashboard weekly update doesn't seem to have happened on 2017-05-28 - https://phabricator.wikimedia.org/T166884#3310804 (10JAllemandou) As of today 2017-06-05, data is present up to 2017-05-28. Having looked at configuration, I think data for 2017-06-04 week should be computed in t... [08:26:37] 10Analytics-Kanban: Update per-domain uniques fresh-sessions computation - https://phabricator.wikimedia.org/T167005#3314230 (10JAllemandou) [08:28:12] (03PS2) 10Joal: Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (https://phabricator.wikimedia.org/T167005) [08:29:50] (03PS3) 10Joal: Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) [08:48:16] joal: merged and restarted pivot [08:48:40] Yay ! [08:49:01] Thanks elukey :) [08:49:17] eqi stat1004 [08:49:19] oops :) [12:22:46] Hi there. Please: is there anyone who is familiar with running RStudio Shiny Server under a web-proxy from a Labs instance? I would highly appreciate to be able to hear an advice or two. Thanks a lot. [12:46:44] 10Analytics-Kanban: Preserve userAgent field in apps schemas - https://phabricator.wikimedia.org/T164125#3314670 (10mforns) Ping @Tbayer :] [13:01:18] 10Analytics, 10Analytics-EventLogging, 10DBA: db1047 has been restarted - needs another restart - https://phabricator.wikimedia.org/T166452#3314678 (10Marostegui) The server went nuts again and got stuck. I am going to leave it catch up (it is 3 days behind) and will attempt the ALTER once more. If it doesn... [13:45:36] (03CR) 10Ottomata: "Should/could we move Webrequest and WebrequestData classes into the core.webrequest package?" (033 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [13:47:59] (03CR) 10Ottomata: [C: 031] Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (https://phabricator.wikimedia.org/T167005) (owner: 10Joal) [13:49:32] (03CR) 10Ottomata: [C: 031] Add sampled-webrequest druid loading oozie jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) (owner: 10Joal) [13:52:09] (03CR) 10Ottomata: [C: 031] Rename last_access_uniques for consistency [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 (owner: 10Joal) [13:55:08] 10Analytics-EventLogging, 10Analytics-Kanban: whitelist multimedia and upload wizard tables - https://phabricator.wikimedia.org/T166821#3314804 (10mforns) [14:11:55] 10Analytics-Cluster, 10Analytics-Kanban: Provision new Kafka cluster(s) with security features - https://phabricator.wikimedia.org/T152015#3314859 (10Ottomata) Requirements for Kafka TLS encryption and authentication: - All Kafka messages must be encrypted over the wire, broker to broker and broker to client.... [14:13:09] 10Analytics-EventLogging, 10Analytics-Kanban: whitelist multimedia and upload wizard tables - https://phabricator.wikimedia.org/T166821#3314862 (10mforns) Hi @JKatzWMF! I've looked into the schemas, and they seem non-sensitive to me, given that we purge the parsed (sanitized) userAgent field from the capsule... [14:20:07] 10Analytics, 10Analytics-EventLogging: whitelist mobileapp reading list schema - https://phabricator.wikimedia.org/T166823#3314876 (10mforns) @JKatzWMF @Tbayer I already commented on this schema in the thread, but for the record: Yes, this schema is non-sensitive once you remove the userAgent field. Actually... [14:20:32] 10Analytics, 10Analytics-EventLogging: whitelist mobileapp reading list schema - https://phabricator.wikimedia.org/T166823#3314883 (10mforns) [14:20:33] 10Analytics-Kanban: Preserve userAgent field in apps schemas - https://phabricator.wikimedia.org/T164125#3222702 (10mforns) [14:36:02] mforns: o/ [14:36:06] elukey, hi! [14:36:14] I'm looking at your patch [14:36:29] hola! Sorry I am still not ready for the data vetting, just completed the patch today [14:36:50] np [14:37:01] hola! [14:37:03] after standup I am going to try to create a script that inserts data and then tomorrow we should be able to test [14:37:11] from what I can see it works as expected [14:37:14] elukey, wanna pair on the script? [14:37:27] hellooo [14:37:42] nono let me do the first version by myself, so I'll think about the testing data :) [14:46:35] 10Analytics-Kanban: Update per-domain uniques fresh-sessions computation - https://phabricator.wikimedia.org/T167005#3315008 (10Nuria) [14:51:56] (03PS12) 10Nuria: UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) [14:52:11] 10Analytics: Pivot access for Goran S. Milovanovic, WMDE Data Analyst - https://phabricator.wikimedia.org/T167028#3315019 (10GoranSMilovanovic) [14:55:35] 10Analytics: fix (or remove) guard job - https://phabricator.wikimedia.org/T167029#3315042 (10Nuria) [14:57:03] ottomata: since francisco is on vacation today, i will finish his patch and we can deploy? [14:57:15] ottomata: teh puppet side of the filter probably can be merged 1st [14:58:14] *the [15:02:22] 10Analytics-Kanban: Update per-domain uniques fresh-sessions computation - https://phabricator.wikimedia.org/T167005#3315061 (10JAllemandou) a:03JAllemandou [15:07:35] nuria_: sure, i added a comment on it [15:26:21] 10Analytics-Kanban: Update per-domain uniques fresh-sessions computation - https://phabricator.wikimedia.org/T167005#3315160 (10Nuria) [15:26:24] 10Analytics, 10Dumps-Generation: https://dumps.wikimedia.org/other/pageviews/ needs a README - https://phabricator.wikimedia.org/T167033#3315161 (10Halfak) [15:27:06] 10Analytics-Kanban, 10Patch-For-Review: Create tagging udf - https://phabricator.wikimedia.org/T164021#3315174 (10Nuria) [15:29:54] we cancled eventbus meeting [15:29:54] i [15:31:24] ottomata: ok, came to tasking and we can talk about camus? [15:31:27] ping mforns [15:31:44] ya [15:31:45] coming [15:33:05] 10Analytics-Kanban: https://dumps.wikimedia.org/other/pageviews/ needs a README - https://phabricator.wikimedia.org/T167033#3315189 (10Nuria) [15:36:47] 10Analytics: fix (or remove) guard job - https://phabricator.wikimedia.org/T167029#3315209 (10Nuria) [15:36:49] 10Analytics, 10Operations: Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 - https://phabricator.wikimedia.org/T166937#3315212 (10Nuria) [15:37:05] 10Analytics, 10Operations: Broken /a/refinery-source/guard/run_all_guards.sh script on stat1002 - https://phabricator.wikimedia.org/T166937#3312286 (10Nuria) p:05Normal>03High [15:38:46] 10Analytics: Browser analytics dashboard weekly update doesn't seem to have happened on 2017-05-28 - https://phabricator.wikimedia.org/T166884#3315230 (10Nuria) a:03mforns [15:41:19] 10Analytics, 10Analytics-Wikistats, 10Editing-Analysis: Bring the Editor Engagement Dashboard back - https://phabricator.wikimedia.org/T166877#3315238 (10Nuria) @Halibutt Here are some dashboards: https://edit-analysis.wmflabs.org/compare/ https://edit-analysis.wmflabs.org/editor-engagement/#projects=eswi... [15:41:44] 10Analytics, 10Editing-Analysis: Bring the Editor Engagement Dashboard back - https://phabricator.wikimedia.org/T166877#3315240 (10Nuria) [15:54:23] 10Analytics: Browser analytics dashboard weekly update doesn't seem to have happened on 2017-05-28 - https://phabricator.wikimedia.org/T166884#3315267 (10mforns) Hi @Jdforrester-WMF The browser data has weekly granularity, so it is computed after the whole week has passed. Actually, it waits for an extra 24 hou... [15:54:40] 10Analytics, 10Analytics-Cluster: Import Kafka messages into HDFS authenticating with TLS/SSL - https://phabricator.wikimedia.org/T166832#3315268 (10Nuria) Tasks: Evaluate whether camus can use newer consumer, if so, proceed this way. Possible issues: * Does newer consumer APi manage offsets for us? How... [15:54:54] 10Analytics, 10Analytics-Cluster: Import Kafka messages into HDFS authenticating with TLS/SSL - https://phabricator.wikimedia.org/T166832#3315269 (10Nuria) p:05Normal>03High [15:56:04] 10Analytics, 10Analytics-Cluster: Import Kafka messages into HDFS authenticating with TLS/SSL - https://phabricator.wikimedia.org/T166832#3309097 (10Nuria) [15:58:18] 10Analytics: Browser analytics dashboard weekly update doesn't seem to have happened on 2017-05-28 - https://phabricator.wikimedia.org/T166884#3315274 (10mforns) 05Open>03declined [15:58:51] 10Analytics: Blog post about druid - https://phabricator.wikimedia.org/T157978#3315275 (10Nuria) [16:00:34] 10Analytics: Browser analytics dashboard weekly update doesn't seem to have happened on 2017-05-28 - https://phabricator.wikimedia.org/T166884#3315288 (10Jdforrester-WMF) Ah, OK. Is there any chance this could be documented in the interface? [16:03:29] 10Analytics, 10Analytics-EventLogging, 10MediaWiki-Vagrant, 10Services (watching): Vagrant git-update error for event logging - https://phabricator.wikimedia.org/T161935#3315301 (10Nuria) [16:04:26] 10Analytics, 10Analytics-EventLogging, 10MediaWiki-Vagrant, 10Services (watching): Vagrant git-update error for event logging - https://phabricator.wikimedia.org/T161935#3147676 (10Nuria) @Pchelolo : depririotizing as we do not think we will have bandwith to troubleshoot this in the near term, let us know... [16:09:58] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3315324 (10Nuria) Since this is the master for eventlogging machine. Can we move the refresh for this host to happen sooner? (ping @jcrespo) https://phabricator.wikime... [16:13:36] 10Analytics-Kanban: Test failures in refinery master - https://phabricator.wikimedia.org/T166334#3315338 (10Nuria) [16:14:27] 10Analytics: Provide top domain and data to truly test superset - https://phabricator.wikimedia.org/T166689#3315342 (10Nuria) [16:17:02] 10Analytics-Kanban, 10DBA, 10Operations, 10ops-eqiad, 10User-Elukey: db1046 BBU looks faulty - https://phabricator.wikimedia.org/T166141#3315357 (10jcrespo) Not really, we have almost decided the goals for Q1, and they are all quite urgent and for hardware that has been already bought. What we can do to... [16:18:13] 10Analytics: Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#3315367 (10Nuria) [16:18:25] 10Analytics: Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#3315384 (10Nuria) p:05Triage>03Normal [16:18:39] mforns: Can you try compiling again? [16:19:04] 10Analytics: Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#3315367 (10Nuria) [16:19:06] 10Analytics, 10EventBus, 10Patch-For-Review, 10Services (watching): Delete stale kafka topics - https://phabricator.wikimedia.org/T163392#3195930 (10Nuria) [16:20:16] 10Analytics, 10Operations, 10User-Elukey: kafkatee's logrotate/syslog default pkg files needs to be removed - https://phabricator.wikimedia.org/T145490#3315393 (10Nuria) [16:21:25] 10Analytics, 10Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#2745858 (10Nuria) We actually renamed 1001 to be thorium. It is no longer a stats box (in name). [16:21:37] 10Analytics, 10Operations: Rename stat100x machines to have misc element names - https://phabricator.wikimedia.org/T149228#3315397 (10Nuria) 05Open>03Resolved [16:22:55] 10Analytics, 10Analytics-Cluster, 10EventBus: Delete stale topics from main Kafka clusters - https://phabricator.wikimedia.org/T149594#3315401 (10Nuria) [16:22:57] 10Analytics, 10EventBus, 10Patch-For-Review, 10Services (watching): Delete stale kafka topics - https://phabricator.wikimedia.org/T163392#3315403 (10Nuria) [16:23:07] 10Analytics, 10Analytics-Cluster, 10EventBus: Delete stale topics from main Kafka clusters - https://phabricator.wikimedia.org/T149594#2757362 (10Nuria) [16:23:09] 10Analytics: Upgrade Kafka on main cluster with security features - https://phabricator.wikimedia.org/T167039#3315406 (10Nuria) [16:26:41] 10Analytics, 10Operations, 10Performance-Team: Update jq to v1.4.0 or higher - https://phabricator.wikimedia.org/T159392#3066309 (10Nuria) We are moving to new stats boxes which we will not have this problem. [16:26:45] 10Analytics, 10Operations, 10Performance-Team: Update jq to v1.4.0 or higher - https://phabricator.wikimedia.org/T159392#3315423 (10Nuria) 05Open>03declined [16:32:19] (03PS1) 10Joal: Add username in path for test [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357232 (https://phabricator.wikimedia.org/T166334) [16:32:21] mforns: do you get an aerror when running refinery-job mvn tests locally (not in 1004) [16:32:25] *an error? [16:33:09] nuria_, since I wiped the computer I did not install mvn all libs needed... but before, I did not get errors [16:33:17] I can try though [16:33:45] usually mvn-test in 1004 [16:34:08] 10Analytics-Kanban, 10Patch-For-Review: Count project-wide unique devices (like *.wikipedia.org) - https://phabricator.wikimedia.org/T143928#3315457 (10Nuria) [16:36:21] (03PS3) 10Nuria: Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (owner: 10Joal) [16:39:58] 10Analytics-Kanban: Rename last_access_uniques to per-domain uniques - https://phabricator.wikimedia.org/T167043#3315501 (10JAllemandou) [16:42:09] 10Analytics-Kanban: Rename last_access_uniques to per-domain uniques - https://phabricator.wikimedia.org/T167043#3315520 (10JAllemandou) a:03JAllemandou [16:42:53] 10Analytics-Kanban, 10Patch-For-Review: Load webrequest raw data into druid so ops can use it for troubleshooting - https://phabricator.wikimedia.org/T166967#3315524 (10JAllemandou) [16:44:11] (03CR) 10Nuria: Correct per-domain unique devices jobs (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (owner: 10Joal) [16:45:12] (03PS3) 10Joal: Rename last_access_uniques for consistency [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 (https://phabricator.wikimedia.org/T167043) [16:46:35] mforns: no, no need, [16:46:41] mforns:i can re try locally [16:46:46] k [16:46:58] ottomata: let me do 1 change to francisco's patch and we can deploy eventlogging? [16:47:50] nuria_: ok, i'm making some lunch, did you see my one comment? [16:48:07] https://gerrit.wikimedia.org/r/#/c/356624/1/eventlogging/utils.py [16:48:38] mforns: Do you mind reviewing/merging the patch of uniques renaming? [16:48:43] ottomata: ya, that is the one i would chnage [16:48:46] (03PS4) 10Joal: Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (https://phabricator.wikimedia.org/T167005) [16:48:56] joal, not at all, will do in 5 [16:49:10] ottomata: with return 'mediawiki' in user_agent.lower(); [16:49:14] ottomata: sooundsgood? [16:49:18] ottomata: soound sgood? [16:49:25] ottomata: argh [16:50:10] + [16:50:10] +1 [16:50:58] ottomata: would you have a minute for me on splitting? [17:06:46] joal, reviewing [17:06:59] (03CR) 10Nuria: [C: 032] Add username in path for test [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357232 (https://phabricator.wikimedia.org/T166334) (owner: 10Joal) [17:07:10] (03CR) 10Nuria: [C: 032] "Tested on 1004, working." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357232 (https://phabricator.wikimedia.org/T166334) (owner: 10Joal) [17:14:03] (03Merged) 10jenkins-bot: Add username in path for test [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357232 (https://phabricator.wikimedia.org/T166334) (owner: 10Joal) [17:16:18] sorry joal was making lunch, yaaaaaa [17:16:21] what's up? [17:16:59] ottomata: I'm still in wonder about spark versus hive [17:17:08] ottomata: batcave for a minute? [17:18:05] coming [17:23:01] 10Analytics, 10Analytics-EventLogging, 10Community-Tech, 10Editing-Analysis, and 2 others: Record an EventLogging event every time a new mainspace page is created - https://phabricator.wikimedia.org/T150369#3315733 (10kaldari) p:05Low>03Normal [17:30:05] (03CR) 10Bearloga: UDF to tag requests (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [17:31:58] (03CR) 10Nuria: "Merging, let's keep an eye on this change as it will increase the offset count." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (https://phabricator.wikimedia.org/T167005) (owner: 10Joal) [17:32:03] (03CR) 10Nuria: [V: 032 C: 032] Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (https://phabricator.wikimedia.org/T167005) (owner: 10Joal) [17:33:28] ottomata: do eventlogging tests run clean for you in master? [17:34:43] (03CR) 10Joal: "see inline" (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [17:35:12] nuria_: i'm not set up to run them right now... :/ but, i think they do! [17:35:21] or, at least, last time i've checked [17:35:24] my vagrant is a little busted [17:35:56] ottomata: mmmm...no i do not think so, let me triple check [17:45:38] 10Analytics, 10Analytics-EventLogging, 10Community-Tech, 10Editing-Analysis, and 2 others: Record an EventLogging event every time a new mainspace page is created - https://phabricator.wikimedia.org/T150369#3315778 (10Ottomata) > Another reason we would like to have this data in EventLogging Just reiterat... [17:47:41] nuria_: can deploy EL whenever you are ready [17:47:43] then we can make puppet change [17:47:54] ottomata: let me see about tests [17:47:57] ottomata: they fail [17:58:37] (03CR) 10Mforns: "LGTM! I'll merge unless you want to change any of those things first." (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 (https://phabricator.wikimedia.org/T167043) (owner: 10Joal) [17:59:08] joal, ^ if you want to change those, otherwise I'll merge as is [18:02:21] milimetric, Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found [18:03:27] 10Analytics, 10Analytics-EventLogging, 10Community-Tech, 10Editing-Analysis, and 2 others: Record an EventLogging event every time a new mainspace page is created - https://phabricator.wikimedia.org/T150369#3315832 (10dbarratt) a:03dbarratt [18:05:02] MaxSem, where [18:05:17] in production [18:05:43] wth.... that's been fine for weeks, on a certain page or where? [18:05:52] (03CR) 10Joal: Rename last_access_uniques for consistency (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 (https://phabricator.wikimedia.org/T167043) (owner: 10Joal) [18:06:02] (03PS4) 10Joal: Rename last_access_uniques for consistency [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 (https://phabricator.wikimedia.org/T167043) [18:06:44] (03PS5) 10Joal: Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (https://phabricator.wikimedia.org/T167005) [18:07:12] (03CR) 10Mforns: [V: 032 C: 032] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356815 (https://phabricator.wikimedia.org/T167043) (owner: 10Joal) [18:07:39] MaxSem: on a certain page or where do you see the error? [18:07:49] milimetric, in logs [18:08:06] mostl ikely happens on startup [18:08:42] hmmm.... pages seem fine and seem to work. I'll file a bug and look at it Thursday if it's not killing anything. Î'm off until Thursday [18:10:05] ottomata: ok, got problem with eventlogging tests, need to change setup of plugings for tests, will be a sec [18:11:17] 10Analytics: Invalid $wgJsonConfigModels['JsonConfig.Dashiki'] array value, 'class' not found - https://phabricator.wikimedia.org/T167054#3315862 (10Milimetric) a:03Milimetric [18:12:12] ok [18:12:19] i'm fighting with vagrant [18:12:32] mforns: just sent the last version of the code [18:12:50] I checked on my testing environment and it seems working as expected [18:12:53] \o/ [18:15:07] ottomata: you can run tox -e py27 outside vagrant [18:15:29] mforns: the script atm creates one row per day, from now() until 365 days ago [18:15:29] ottomata: as long as you have pip install . and have the right version of liberty kafka [18:15:46] ottomata: https://github.com/confluentinc/confluent-kafka-python/issues/6 [18:16:02] mforns: so it is easy to check the id attribute to match older/newer than values [18:16:06] ottomata: otherwise you get bizarre errors that happen cause libertykafka is too old [18:16:24] haha libertykakfa [18:16:26] mforns: let's vet the data tomorrow and then possibly do it in beta together? [18:16:35] ottomata: i just had never looked at the plugging code so doing that now so i can best fix issue [18:18:22] ottomata: what happens is that plugings are being initialized on test run twice [18:18:27] going offline team! See you tomorrow :) [18:18:29] * elukey off! [18:18:34] ottomata: and thus run into an error of alredy redefined [18:18:43] hm ok [18:18:49] ottomata: some teardown file needs to "undefine" the plugings [18:20:17] nuria_: still trying to get a runnable el locally... [18:20:30] ottomata: ya it tooke me a while too [18:27:58] vagrant mostly works for me, but it won't mount the /vagrant directory anymore [18:28:11] and if i try to recreate teh vm, it never boots [18:28:20] trying docker... [18:29:21] ottomata: the issue is https://github.com/wikimedia/eventlogging/blob/master/eventlogging/handlers.py#L57 [18:29:38] ottomata: as it raises an exception if a 'plugin' module is redefined [18:29:52] alllright [18:29:54] ottomata: tests initialize plugins twice [18:29:55] i think i've got it! [18:30:00] ottomata: and thsu run into that issue [18:30:00] jyeah, seeing plugin test error [18:30:01] looking [18:30:19] oh because each tests reloads the module? [18:30:20] hmm [18:30:30] ottomata: two tests load plugins [18:30:39] ottomata: one this one by ori a loong time ago [18:30:42] ahhh ok [18:30:43] got it [18:30:48] i can fix [18:31:05] ottomata: https://github.com/wikimedia/eventlogging/blob/master/tests/test_handlers.py#L55 [18:31:45] ottomata: other a stream tests we added: https://github.com/wikimedia/eventlogging/blob/master/tests/test_streams.py#L41 [18:32:07] ottomata: what shoudl be right is that teardown cleans plugin loading [18:32:29] yup [18:32:42] ottomata: but man ... https://docs.python.org/2/library/imp.html doesn't look like there is a "supported" way to do it [18:33:05] yeahhh, but we just need tests to bypass the error, which is not considering what is currently imported... [18:33:42] nuria_: https://gerrit.wikimedia.org/r/#/c/357237/ [18:34:11] ottomata: ah nice! [18:35:28] ottomata: but you need to do that in setUp [18:35:42] ottomata: BEFORE loading, makes sense? [18:36:14] ottomata: so setup and teardown [18:36:49] why? [18:37:10] https://github.com/wikimedia/eventlogging/blob/master/eventlogging/handlers.py#L42 [18:38:19] ottomata: because teardown runs after the test so if any other test initializes pluggings before you that one test will fail, right? [18:39:32] hmmmmm [18:39:51] ya suppose so [18:39:54] ok [18:41:05] ottomata: will send patch [18:42:02] done already [18:42:06] ottomata: k [18:42:07] nuria_: ^ [18:42:49] ottomata: ok, let's merge this one [18:42:52] k [18:46:27] ottomata: how did we bypassed py3 errors on jenkins before? [18:48:33] remove jenkins-bot as a reviewer [18:49:47] 10Analytics, 10Analytics-EventLogging, 10Editing-Analysis, 10Wikimedia-Hackathon-2017, and 2 others: Record an EventLogging event every time a new mainspace page is created - https://phabricator.wikimedia.org/T150369#3316059 (10dbarratt) [18:56:21] ottomata: ok let me fix tox issues [18:56:26] ottomata: sorry flake8 [18:57:03] ottomata: k, merged [18:59:08] coo [19:02:33] ottomata: ok, rebased and merged francisco's patch [19:02:46] ottomata: did you merged the other puppet patch [19:02:48] ? [19:02:57] puppet patch? [19:05:47] ottomata: https://gerrit.wikimedia.org/r/#/c/356626/ [19:06:16] ah didn't see that one, nuria_ i want to modify that, since it isn't really is_not_bot anymore [19:06:21] nuria_: you've deploye dalready? [19:06:28] ottomata: no no [19:06:47] ottomata: i just rebased and merged is_mediawiki patch [19:07:15] ottomata: argh , but wait , i did not pushed my last change to francisco's patch , ayayayay [19:07:33] ottomata: to do the mediawiki in ua.lowercase() [19:08:08] ottomata: super sorry about that [19:09:15] np [19:09:20] ottomata: will do change [19:09:27] just ammended https://gerrit.wikimedia.org/r/#/c/356626/ [19:09:29] you can review that [19:09:50] hmm, wait, no, that logic isn't right [19:10:52] ottomata: k [19:13:53] nuria_: https://gerrit.wikimedia.org/r/#/c/356626/ ok [19:16:33] nuria_: i'm going to run in 30 mins, shall we make this happen before t hen? [19:16:51] ottomata:we can do it after if you are available this afternoon, fine with me [19:17:08] naw, i'm taking the rest of the afternoon off since i worked friday morning [19:17:23] gonna go trim some stuff at the ceramics studio! got class tonight :) [19:19:44] ottomata: ok, then let's do it tomorrow morning [19:19:46] ok [19:37:58] ottomata: also fixed my merge: https://gerrit.wikimedia.org/r/#/c/357243/1/eventlogging/utils.py [19:46:24] 10Analytics-Kanban: Improve Oozie error emails for testing - https://phabricator.wikimedia.org/T161619#3316236 (10Nuria) Updated: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Oozie#Make_sure_your_test_run_is_not_spaming_everyone_with_e-mails [19:46:43] nuria_: semi colon not needed [19:47:17] ottomata: indeed, python will never sync on my brain [20:18:36] 10Analytics, 10Analytics-EventLogging, 10Editing-Analysis, 10Wikimedia-Hackathon-2017, and 2 others: Record an EventLogging event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3316379 (10kaldari) [20:49:00] (03CR) 10Nuria: [V: 032 C: 032] Correct per-domain unique devices jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/356823 (https://phabricator.wikimedia.org/T167005) (owner: 10Joal) [21:01:10] (03CR) 10Nuria: UDF to tag requests (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [21:01:43] bearloga: please see my comment on https://gerrit.wikimedia.org/r/#/c/353287/12 regarding portal tagging [21:09:24] bearloga: i am also of the opinion that a "portal" tag is to be preffered, let me know if you disagree, do you count those pageviews separately and that is why you wanted separated tags? [21:10:14] (03PS13) 10Nuria: UDF to tag requests [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) [21:10:41] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun 2017): Automatically sync mediawiki-identities/wikimedia-affiliations.json DB dump file with the data available on wikimedia.biterg.io - https://phabricator.wikimedia.org/T157898#3316525 (10Aklapper) >>! In T157898#3310762, @Albertinisg wrote:... [21:12:14] (03CR) 10Nuria: ">Should/could we move Webrequest and WebrequestData classes into the core.webrequest >package?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [21:16:31] 10Analytics-Kanban, 10Patch-For-Review: Improve processing of host on refine step on Webrequest.java - https://phabricator.wikimedia.org/T166628#3316536 (10Nuria) 05Open>03Resolved [21:16:50] 10Analytics-Kanban: Evaluate swiv and see whether outstanding pivot bugs are fixed - https://phabricator.wikimedia.org/T166320#3316537 (10Nuria) 05Open>03Resolved [21:17:11] 10Analytics-Kanban, 10Patch-For-Review: Refactor monthly banner oozie job to use already indexed daily data - https://phabricator.wikimedia.org/T159727#3316539 (10Nuria) 05Open>03Resolved [21:17:23] 10Analytics-Kanban: Improve Oozie error emails for testing - https://phabricator.wikimedia.org/T161619#3316540 (10Nuria) 05Open>03Resolved [21:18:18] 10Analytics-EventLogging, 10Analytics-Kanban: whitelist multimedia and upload wizard tables - https://phabricator.wikimedia.org/T166821#3316542 (10Nuria) [21:18:26] 10Analytics-EventLogging, 10Analytics-Kanban: whitelist multimedia and upload wizard tables - https://phabricator.wikimedia.org/T166821#3308710 (10Nuria) [21:22:00] 10Analytics-EventLogging, 10Analytics-Kanban: whitelist multimedia and upload wizard tables - https://phabricator.wikimedia.org/T166821#3316551 (10JKatzWMF) @mforns Works for me. Thanks! [21:22:02] 10Analytics-Kanban: Correct pageview_hourly loading scheme on pivot homescreen - https://phabricator.wikimedia.org/T167068#3316552 (10Nuria) [21:22:10] 10Analytics-Kanban: Correct pageview_hourly loading scheme on pivot homescreen - https://phabricator.wikimedia.org/T167068#3316566 (10Nuria) [21:56:37] (03CR) 10Bearloga: UDF to tag requests (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [22:23:38] (03PS1) 10Nuria: Specifying test dependency on snappy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357318 [22:26:31] (03PS2) 10Nuria: Specifying test dependency on snappy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357318 [22:30:06] (03PS1) 10Nuria: Revert "Specifying test dependency on snappy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357322 [22:30:35] (03CR) 10Nuria: [V: 032 C: 032] Revert "Specifying test dependency on snappy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357322 (owner: 10Nuria) [22:34:30] (03PS1) 10Nuria: Specifying test dependency on snappy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/357324 [22:55:56] (03CR) 10Nuria: UDF to tag requests (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/353287 (https://phabricator.wikimedia.org/T164021) (owner: 10Nuria) [22:59:20] hi you all. Is there someone from Analytics around? :) [23:22:14] leila: I am here if for a (very ) brief time [23:24:11] hi nuria_. not sure how brief it will be. feel free to drop it for later if it's not brief. The question is: do you have plans to share publicly data about the usage of Wikidata items, statements, etc.? We have a request T143819 and I do know that this kind of information is useful for many institutions that do data donation to WD, and I'm wondering: are there plans to solve this problem once and for all, for everyone? :) And if there is none atm [23:24:11] T143819: Data request for logs from SparQL interface at query.wikidata.org - https://phabricator.wikimedia.org/T143819 [23:24:14] ping leila [23:39:26] nuria_: ^