[07:40:19] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Asynchronous processing in production: one queue to rule them all - https://phabricator.wikimedia.org/T149408#2980990 (10Joe) https://commons.wikimedia.org/wiki/File:Asynchronous_processing_on_the_WMF_cluster.pdf is the uploaded file.
[07:47:05] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 5 others: Asynchronous processing in production: one queue to rule them all - https://phabricator.wikimedia.org/T149408#2980993 (10Joe) 05Open>03Resolved
[08:56:58] <elukey>	 joal: o/
[08:57:10] <elukey>	 if you are ok I'd bootstrap aqs1007-b! (https://gerrit.wikimedia.org/r/#/c/334753)
[09:09:16] <wikibugs>	 10Analytics-Tech-community-metrics, 06Developer-Relations (Jan-Mar-2017): Merge detached Phab and mw.org identities in korma DB if Phab API shows that accounts are linked - https://phabricator.wikimedia.org/T156216#2981229 (10Aklapper) 05Open>03Resolved ``` MariaDB [sortinghat_wm]> select mwid, phabuuid fr...
[09:13:53] <joal>	 Hi elukey !
[09:13:58] <joal>	 let's do that !
[09:14:08] <elukey>	 gogogo
[09:14:11] <joal>	 elukey: today is tough day for me: coffee machine broke :(
[09:14:24] <elukey>	  /o\
[09:14:49] <joal>	 indeed - There are moments when living in the country-side is not helping
[09:15:05] <elukey>	 http://giphy.com/search/noooo
[09:15:19] <joal>	 :)
[09:15:33] <joal>	 It'll just take me a few more hours to try to put my brain on
[09:17:19] <wikibugs>	 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2981235 (10ArielGlenn) Rather than letting this die, is there any sort of concensus on next steps here?
[09:35:42] <elukey>	 joal: aqs1007-b boostrapping!
[09:36:10] <joal>	 Yay ! it even bootstraps from 1007-a (same rack !)
[09:37:10] <elukey>	 yesss!
[09:37:48] <elukey>	 joal: did you read Eric's investigation about why the aqs1007-a was slow to boostrap?
[09:38:00] <joal>	 nope, not seen it
[09:38:02] <elukey>	 receiving threads were at 100%
[09:38:08] <joal>	 makes sense :)
[09:38:13] <elukey>	 probably due to compression related ops
[09:38:13] <joal>	 multi threading?
[09:38:25] <joal>	 cause the machine was not overloaded
[09:38:51] <elukey>	 he told me to use 
[09:38:51] <joal>	 clearer: could we give cassandra more receiving threads for that moment?
[09:38:52] <elukey>	 sjk ttop -s localhost:`uyaml /etc/cassandra-instances.d/aqs1007-a.yaml /jmx_port` -f STREAM*
[09:39:08] <elukey>	 WOW this time it is much faster
[09:39:09] <elukey>	 ahhahah
[09:39:25] <elukey>	 aqs1007-a.eqiad.wmnet      110.3GiB    109.0GiB   98.77%   12.3MiB/s   2.51h
[09:39:25] <joal>	 wow, it is indeed
[09:39:28] <elukey>	 go gogogo
[09:40:27] <joal>	 elukey: a fun thing is the disk utilisation getting way higher from 1007-a bootstrap :)
[09:42:47] <wikibugs>	 (03CR) 10Joal: [C: 031] "LGTM, let's confirm it has been tested, and merge :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns)
[09:47:38] <elukey>	 now the transfer rate is stabilized even from aqs1007-a
[09:47:41] <elukey>	 [000213] user=99.67% sys= 0.27% alloc=  178mb/s - STREAM-IN-/10.64.0.213
[09:47:44] <elukey>	 [000216] user=99.67% sys= 0.28% alloc=  180mb/s - STREAM-IN-/10.64.0.126
[09:47:47] <elukey>	 [000218] user=99.86% sys= 0.04% alloc=  200mb/s - STREAM-IN-/10.64.0.127
[09:48:12] <elukey>	 anyhow, not in a hurry
[09:48:22] <elukey>	 tomorrow morning aqs1007-b should be ok
[09:48:29] <elukey>	 brb!
[10:01:13] <icinga-wm>	 PROBLEM - cassandra-b CQL 10.64.0.237:9042 on aqs1007 is CRITICAL: connect to address 10.64.0.237 and port 9042: Connection refused
[10:01:27] <joal>	 elukey: can you acknowledge that -^
[10:09:58] <elukey>	 here I am!
[10:11:56] <icinga-wm>	 ACKNOWLEDGEMENT - cassandra-b CQL 10.64.0.237:9042 on aqs1007 is CRITICAL: connect to address 10.64.0.237 and port 9042: Connection refused Elukey bootstrapping cassandra
[10:12:04] <joal>	 Thanks :)
[10:28:49] <wikibugs>	 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#2981471 (10elukey) I followed up with Sam via chat, we are following up with Legal to verify NDA compatibility with this request before proceeding any further.
[10:57:27] <wikibugs>	 10Analytics-Tech-community-metrics, 06Labs: http://korma.wmflabs.org/ got erased - https://phabricator.wikimedia.org/T156253#2981529 (10Aklapper) a:03Aklapper
[10:57:54] <wikibugs>	 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#2981531 (10jrobell) thank you @elukey. I started an email thread with you and James at legal who has been handling the contracts with CPS for us.
[11:09:20] <wikibugs>	 10Analytics: Current pageview definition leads to unwanted statistics lifecycle - https://phabricator.wikimedia.org/T156628#2981543 (10JAllemandou)
[11:12:37] <wikibugs>	 06Analytics-Kanban: Better explanation on pageview definition for edit actions - https://phabricator.wikimedia.org/T156629#2981561 (10JAllemandou)
[11:18:37] <wikibugs>	 (03PS1) 10Joal: Add comment to action=edit filter in pageview def [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/335002 (https://phabricator.wikimedia.org/T156629)
[11:23:22] <wikibugs>	 10Analytics: Current pageview definition leads to unwanted statistics lifecycle - https://phabricator.wikimedia.org/T156628#2981583 (10Trizek-WMF)
[13:14:43] <elukey>	 so if things will go as planned, tomorrow morning aqs1007-a/b should be fully bootstrapped and we could run nodetool cleanup on aqs1004
[13:15:38] <elukey>	 then bootstrap aqs1008-a, wait a day, bootstrap aqs1008-b and run nodetool cleanup on aq1005 (basically ~EOD Thursday)
[13:15:56] <elukey>	 then start aqs1009-a/b, that should be completed ~ next monday
[13:16:23] <elukey>	 final step adding aqs100[789] to LVS to serve traffic
[13:17:14] <joal>	 elukey: agreed on the sequence of events, not on calendar time - contingency rules tell me it probably won't be done before end of next week ;)
[13:20:11] <elukey>	 joal: yes yes this is the optimistic schedule :P
[13:20:25] * fdans is out for 20 to get lunch :)
[13:48:32] <joal>	 taking a break a-team, later !
[14:14:41] <elukey>	 going to the office (a bit late), will brb in a few!
[14:21:24] <urandom>	 elukey: \o
[14:21:49] <urandom>	 elukey: before i forget, you'll want to do a cleanup on the nodes of rack1 after the current bootstrap finishes
[14:22:21] <urandom>	 elukey: you can skip 1007-b (though it wouldn't hurt anything to run it there, it'll just complete right away without doing anything)
[14:23:02] <urandom>	 elukey: if it's impacting (probably won't be, but...), you can throttle compaction throughput, see: c-foreach-nt getcompactionthoughput
[14:23:34] <urandom>	 elukey: afterward, disk space usage will reflect your post-bootstrap reality
[14:38:30] <elukey>	 urandom: o/
[14:38:51] <elukey>	 I wanted to ask to you some info about clenaups :
[14:38:53] <elukey>	 :)
[14:39:46] <wikibugs>	 (03PS1) 10Gerrit Patch Uploader: Amend name for Goan Konkani [analytics/wikipagestats] - 10https://gerrit.wikimedia.org/r/335030 (https://phabricator.wikimedia.org/T126148)
[14:39:48] <wikibugs>	 (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [analytics/wikipagestats] - 10https://gerrit.wikimedia.org/r/335030 (https://phabricator.wikimedia.org/T126148) (owner: 10Gerrit Patch Uploader)
[14:40:13] <wikibugs>	 (03CR) 10Mforns: [C: 032] Add comment to action=edit filter in pageview def [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/335002 (https://phabricator.wikimedia.org/T156629) (owner: 10Joal)
[14:41:00] <elukey>	 urandom: should I just run nodetool-{a,b} cleanup on aqs1004 after the aqs1007-b boostrap finishes right? 
[14:41:22] <elukey>	 afaiu it will drop data from all the keyspaces that are not managed anymore by the instance
[14:41:59] <elukey>	 so in this case, aqs1004-{a,b} "lost" responsibility for the keys now handled by aqs1007-a and soon aqs1007-b
[14:42:15] <wikibugs>	 (03CR) 10Mforns: "> LGTM, let's confirm it has been tested, and merge :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns)
[14:42:21] <elukey>	 no other instance has been impacted other than those because of the rack awareness and replication 3 factor
[14:42:29] <elukey>	 am I vaguely close to the truth?
[14:44:32] <wikibugs>	 (03Merged) 10jenkins-bot: Add comment to action=edit filter in pageview def [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/335002 (https://phabricator.wikimedia.org/T156629) (owner: 10Joal)
[14:56:29] <wikibugs>	 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2981937 (10chasemp)  > - Parallelization is limited by user-cuncurrent-connection limit (10 currently).  @jcrespo /@chasemp - Is this a negociable parameter? >   Open to helping det...
[15:05:45] <wikibugs>	 06Analytics-Kanban, 13Patch-For-Review: Run a 1-off sqoop over the new labsdb servers - https://phabricator.wikimedia.org/T155658#2981969 (10elukey) >>! In T155658#2973803, @JAllemandou wrote: >  > @elukey: Quick question about network: Is the hole opened potentially to be open forever, or will be it be patch...
[15:07:29] <urandom>	 elukey: correct
[15:07:43] <urandom>	 elukey: and this is one of those cases made easier to grok because of that parity
[15:08:03] <urandom>	 you know where the data came from, so you can start your cleanups with confidence while you move onto another rack
[15:08:23] <urandom>	 elukey: also, i think you need to sudo as the cassandra user to do that cleanup
[15:08:56] <urandom>	 elukey: this is a 2.2.x ism, an issue of permissions since it reads from the commitlog now
[15:09:18] <urandom>	 elukey: sudo -u cassandra nodetool-x cleanup
[15:10:33] <elukey>	 urandom: nice! thanks! 
[15:10:40] <urandom>	 elukey: it will block
[15:10:44] <urandom>	 btw
[15:10:55] <elukey>	 block?
[15:10:59] <urandom>	 it blocks so that the command can give you feedback about any progress
[15:11:06] <urandom>	 yeah, nodetool blocks until the cleanup completes
[15:11:09] <urandom>	 which can be days
[15:11:15] <elukey>	 ahhh okok! I thought the instance
[15:11:19] <urandom>	 if you ctrl-c out of it, the cleanup will continue
[15:11:34] <elukey>	 I'll start a tmux session and that's it :)
[15:11:39] <elukey>	 so it may last for days?
[15:11:57] <urandom>	 what i do is to run it with screen, and i invoke it as a command to screen, so the session goes away when the cleanup is done
[15:12:14] * elukey nods
[15:12:19] <urandom>	 that why i can screen -ls to see if they're still running
[15:12:22] <urandom>	 elukey: yeah, it can
[15:12:27] <urandom>	 depends on compaction throughput
[15:12:50] <urandom>	 it'll definitely run for more than "minutes"
[15:12:53] <urandom>	 hours at least
[15:13:28] <urandom>	 elukey: it has to rewrite every sstable that contains data sent to new instances (count on that being all of them)\
[15:13:33] <elukey>	 because I'd prefer not to bootstrap anything during that time window JUST IN CASE TM, even though I got the part that it should be super fine
[15:13:43] <urandom>	 it's very safe
[15:14:11] <urandom>	 worst case it will create some IO, probably not very much, if you have *any* cieling there you should be fine
[15:14:22] <urandom>	 it limits the number of cleanup compactors running at once
[15:14:34] <urandom>	 and even then, you can always lower the compaction throughput
[15:14:59] <urandom>	 in fact, i would argue that *if* it had any impact, it's only because your compaction throughput isn't adequately throttled already
[15:15:45] <urandom>	 and your next bootstrap will be in another rack against an entirely different set of nodes
[15:17:00] <urandom>	 elukey: by safe i mean, all it will do is rewrite an sstable sans the affected keys, and once it has been completely rewritten, it'll atomically replace the source table
[15:17:14] <urandom>	 elukey: even if it failed spectacularly, it won't hurt anything
[15:18:50] <urandom>	 elukey: another way to look at it is that it's just a slightly different compaction use-case, and no more dangerous
[15:20:00] <elukey>	 super
[15:20:28] <elukey>	 our limits are
[15:20:28] <elukey>	 cassandra::compaction_throughput_mb_per_sec: 256
[15:20:28] <elukey>	 cassandra::concurrent_compactors: 12
[15:20:28] <elukey>	 cassandra::concurrent_writes: 64
[15:20:29] <elukey>	 cassandra::concurrent_reads: 64
[15:21:17] <elukey>	 so there should be some fences to prevent a cleanup to go awol
[15:24:45] <urandom>	 you can always tinker with: nodetool-n setcompactionthroughput <i>
[15:29:14] <elukey>	 yep! 
[15:29:22] <elukey>	 urandom: thanks for all the info! :)
[15:31:56] <urandom>	 elukey: sure!
[15:36:13] <wikibugs>	 (03PS2) 10Fdans: [wip] Adds map visualizer to Dashiki [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/333922 (https://phabricator.wikimedia.org/T153921)
[15:36:34] <milimetric>	 fdans: you'll remove the wip when you want me to take another look at that, right?
[15:38:45] <fdans>	 milimetric: yep!
[15:39:38] <milimetric>	 k, cool, just making sure I'm not holding you up
[15:40:38] <fdans>	 milimetric right now the only thing is that the update logic is a bit dumb 
[16:00:58] <nuria>	 joal: standduppp
[16:04:54] <wikibugs>	 10Analytics-Tech-community-metrics, 07Upstream: Make Perceval not index IP address accounts (and consider removing MediaWiki accounts from our DB) - https://phabricator.wikimedia.org/T149482#2982161 (10Aklapper)
[16:04:56] <wikibugs>	 10Analytics-Tech-community-metrics, 07Upstream: When indexing new users, identify identical email addresses and merge identities accordingly in the DB - https://phabricator.wikimedia.org/T151634#2982162 (10Aklapper)
[16:11:34] <wikibugs>	 10Analytics-Tech-community-metrics, 07Upstream: Make Perceval not index IP address accounts (and consider removing MediaWiki accounts from our DB) - https://phabricator.wikimedia.org/T149482#2982188 (10Nemo_bis)
[16:40:49] <joal>	 Hey mforns - Should we wait for an answer from AndyRussG on the TODO, or move forward?
[16:41:10] <joal>	 meaning: Should I merge your patch?
[16:41:13] <AndyRussG>	 Hi joal and mforns ... Which TODO?
[16:41:35] <AndyRussG>	 On the Pivot patch I guess? (checking quickly... sorry for any delay)
[16:42:14] <joal>	 yes AndyRussG, in druid files, goecoded_consistent field
[16:43:37] <AndyRussG>	 K I see it... Replying... :)
[16:43:54] <joal>	 Thanks AndyRussG (we're in a meeting, we'll review after)
[16:45:49] <lzia>	 hey nuria. I noticed you moved https://phabricator.wikimedia.org/T153923 to paused. Does your team still need feedback from research or the pause applies to getting that feedback, too?
[16:46:21] <joal>	 Hi lzia, we're grooming our board, expect an answer in an hour or so ;)
[16:46:33] <lzia>	 got you, thanks, joal
[16:46:35] <lzia>	 :)
[16:48:29] <wikibugs>	 10Analytics, 06Discovery, 06Discovery-Analysis (Current work), 03Interactive-Sprint, 13Patch-For-Review: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#2982368 (10Nuria)
[16:48:39] <AndyRussG>	 joal: yw, thank u :)
[16:49:22] <milimetric>	 lzia: we're pausing until you decide how you want to work on it, doesn't mean we're not working on it
[16:49:30] <milimetric>	 (the in-progress was just too long)
[16:51:34] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: EventLogging fails to validate a Recentchanges event for he.wikipedia.org - https://phabricator.wikimedia.org/T154395#2982387 (10Nuria) 05Open>03Resolved
[16:52:53] <wikibugs>	 10Analytics, 10Analytics-Dashiki: Improve initial load performance for dashiki dashboards - https://phabricator.wikimedia.org/T142395#2982391 (10Nuria) a:05Nuria>03None
[16:56:00] <lzia>	 I see. Thanks, milimetric.
[16:56:12] <wikibugs>	 (03CR) 10AndyRussG: Add banner activity oozie jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns)
[16:57:40] <wikibugs>	 10Analytics, 10Analytics-Dashiki: Switch to fetch away from jquery - https://phabricator.wikimedia.org/T148053#2982401 (10Nuria)
[16:58:39] <wikibugs>	 10Analytics, 10Analytics-Dashiki: Bookmarkable date filters for browser stats dashboard - https://phabricator.wikimedia.org/T143689#2982403 (10Nuria)
[17:03:16] <wikibugs>	 06Analytics-Kanban, 10Pageviews-API, 06Reading-analysis: Skewed pageviews for Azerbaijani and Bulgarian Wikipedias, September, October and November 2016 - https://phabricator.wikimedia.org/T153699#2982432 (10JAllemandou) a:03JAllemandou
[17:19:36] <wikibugs>	 10Analytics, 10Analytics-EventLogging: Remove ad-hoc UA logging from existing schemas - https://phabricator.wikimedia.org/T61832#2982458 (10Nuria) Let's just do a bit of an audit to see what schemas are being used and ping owners. Likely old these are no longer used. This ticket is 2 years old.
[17:21:17] <wikibugs>	 10Analytics-Dashiki, 06Analytics-Kanban, 13Patch-For-Review: Migrate from bower to npm and clean up folder hierarchy - https://phabricator.wikimedia.org/T147884#2982465 (10Nuria)
[17:28:53] <wikibugs>	 10Analytics-General-or-Unknown, 06Analytics-Kanban: analytics.wikimedia.org loads resources from third parties - https://phabricator.wikimedia.org/T156347#2982488 (10Nuria)
[17:29:52] <wikibugs>	 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#1922131 (10Ocaasi_WMF) @Legoktm @kaldari @Milimetric @Krenair Hi folks.  We're having a hard t...
[17:33:57] <wikibugs>	 10Analytics, 10Analytics-EventLogging: Remove ad-hoc UA logging from existing schemas - https://phabricator.wikimedia.org/T61832#2982493 (10Jdlrobson)
[17:36:16] <wikibugs>	 10Analytics, 10ChangeProp, 10EventBus, 06Revision-Scoring-As-A-Service, and 2 others: Rewrite ORES precaching change propagation configuration as a code module - https://phabricator.wikimedia.org/T148714#2982530 (10Nuria) Ping @Ottomata any ideas here?
[17:39:26] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 06Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: EL unable to decode mobile events due to appinstallid - https://phabricator.wikimedia.org/T96940#2982538 (10Nuria) a:03Nuria
[17:40:02] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 06Wikipedia-iOS-App-Backlog, 10iOS-app-Bugs: EL unable to decode mobile events due to appinstallid - https://phabricator.wikimedia.org/T96940#1229822 (10Nuria) 05Open>03Resolved
[17:41:38] <wikibugs>	 10Analytics, 10Analytics-General-or-Unknown: Number of Wikipedia Zero increasing drastically in mid March 2014 - https://phabricator.wikimedia.org/T64848#672451 (10Nuria) Sorry, this ticket is quite old and refers to an infrastructure we no longer use to count pageviews for zero . Closing.
[17:41:47] <wikibugs>	 10Analytics, 10Analytics-General-or-Unknown: Number of Wikipedia Zero increasing drastically in mid March 2014 - https://phabricator.wikimedia.org/T64848#2982576 (10Nuria) 05Open>03Resolved
[17:42:48] <wikibugs>	 (03PS1) 10Milimetric: Bring libraries locally [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/335059 (https://phabricator.wikimedia.org/T156347)
[17:43:05] <milimetric>	 nuria: that's the patch ^
[17:43:29] <milimetric>	 if you wanna merge, I'll write up a question about the proposed attack I was thinking
[17:43:29] <nuria>	 joal: let's move 1 on 1 so we have full 30 mins to wed, ok?
[17:44:23] <joal>	 k nuria 
[17:44:36] <fdans>	 milimetric do you mind holding off the CR until tomorrow? there's a couple things I want to add first
[17:44:49] <wikibugs>	 (03CR) 10Nuria: [V: 032 C: 032] "Looks good, thank you for doing changes." [analytics/analytics.wikimedia.org] - 10https://gerrit.wikimedia.org/r/335059 (https://phabricator.wikimedia.org/T156347) (owner: 10Milimetric)
[17:44:50] <mforns>	 thanks AndyRussG for the comment, so you guys think we should go ahead and deploy as is? cc: joal
[17:45:06] <milimetric>	 np fdans, anytime
[17:45:17] <AndyRussG>	 mforns: I'm just trying out some related queries btw
[17:45:20] <fdans>	 :) thank you
[17:45:26] <mforns>	 AndyRussG, OK
[17:45:45] <AndyRussG>	 mforns: do you remember what day/hour we were running the test queries on?
[17:46:07] <milimetric>	 nuria: one small non-obvious thing from that patch, the semantic.min.css needed the themes folder for the icons, so I changed the download to access it from ../dashboards/themes
[17:46:29] <milimetric>	 I should've added a note or README, my bad
[17:46:29] <mforns>	 AndyRussG, mmmmm no...
[17:46:43] <AndyRussG>	 So far the field actually seems OK now... trying now with each value wrapped in a trim() to see if results vary
[17:47:26] <AndyRussG>	 I guess it probably wouldn't hurt to add it... Just one more dimension to split on, is all
[17:47:35] <mforns>	 AndyRussG, are you looking at the table we created in your own db in hive?
[17:47:51] <AndyRussG>	 mforns: no, I'm querying directly wmf.webrequest
[17:47:59] <mforns>	 oh
[17:48:27] <AndyRussG>	 I'll post the query in a gerrit comment?
[17:49:36] <mforns>	 AndyRussG, sure (it doesn't contain any IPs or UserAgents right?)
[17:52:08] <wikibugs>	 10Analytics-General-or-Unknown, 06Analytics-Kanban, 13Patch-For-Review: analytics.wikimedia.org loads resources from third parties - https://phabricator.wikimedia.org/T156347#2971655 (10Milimetric) Thanks for spotting, @Nemo_bis.  Fixed in the patch above, but I wanted to put down a possible attack and think...
[17:53:18] <AndyRussG>	 nope
[17:53:50] <wikibugs>	 06Analytics-Kanban, 06Operations: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#2982604 (10Milimetric) Thanks, @jcrespo, I didn't see the ping, it looks like Phabricator had some notification issues.  The idea with this service is that it wouldn't take time away from ops, s...
[17:55:17] <wikibugs>	 (03CR) 10AndyRussG: Add banner activity oozie jobs (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/331794 (https://phabricator.wikimedia.org/T155141) (owner: 10Mforns)
[17:55:37] <AndyRussG>	 mforns: K commented...
[17:55:44] <mforns>	 thanks AndyRussG !
[17:55:48] <AndyRussG>	 likewise!
[17:56:01] <AndyRussG>	 I wonder what it was that was not working before...
[17:58:06] <mforns>	 AndyRussG, makes sense, I'll add that field to the job, however, I fear that the 29 of november won't have this value because the corresponding webrequest data will have been discarded by now...
[17:58:38] <mforns>	 not sure about Nov 30th
[17:59:10] <wikibugs>	 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#2982610 (10Nuria) @Ocaasi_WMF: Analytics can help troubleshoot eventlogging issues as needed b...
[18:03:05] * milimetric lunching
[18:04:44] <mforns>	 nov 30th still there
[18:07:15] <AndyRussG>	 mforns: that's fine...! thx!
[18:11:14] <wikibugs>	 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2982645 (10Tgr) The application that creates pageview dumps should escape or filter newlines (and tabs and other unexpected charact...
[18:12:15] <wikibugs>	 10Analytics, 06Discovery, 06Discovery-Analysis (Current work), 03Interactive-Sprint, 13Patch-For-Review: Add Maps tile usage counts as a Data Cube in Pivot - https://phabricator.wikimedia.org/T151832#2982660 (10debt) Hi @Nuria, @mpopov would be the best person to answer this question, but he's on jury du...
[18:14:48] <wikibugs>	 06Analytics-Kanban, 06Operations: Periodic 500s from piwik.wikimedia.org - https://phabricator.wikimedia.org/T154558#2982674 (10jcrespo) I recently packaged and puppetized [[ http://proxysql.com/ | ProxySQL ]]: https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/proxysql/manifests/init.p...
[18:19:46] <elukey>	 going afk team!
[18:19:48] <elukey>	 o/
[18:23:06] <AndyRussG>	 mforns: fwiw there are a quite a few imaginable ways we could further develop the banner pivot tool... For example, WRT geo consistency, we could add as a measure the ratio of geo-consistent vs. geo-inconsistent requests in each segment (is "segment" the right word?)
[18:24:42] <mforns>	 AndyRussG, you mean segment=minute?
[18:24:43] <AndyRussG>	 Other stuff could involve UIs, maybe that could pull a campaign's specifications from a CentralNotice API and show what percentage of targeted pageviews a campaign was recieving
[18:25:11] <AndyRussG>	 no, I mean unique permutation of values for all dimensions
[18:25:32] <AndyRussG>	 Hmmm actually maybe that wouldn't do it
[18:25:53] <mforns>	 AndyRussG, I see, segment is fine I guess, we call them buckets sometimes
[18:25:59] <wikibugs>	 06Analytics-Kanban, 10Pageviews-API: Monthly aggregate endpoint returns unexpected results and invalid timestamp - https://phabricator.wikimedia.org/T156312#2982730 (10Nuria) a:05Nuria>03fdans
[18:26:24] <nuria>	 a-team: per fdans request I am assigning him our outstanding bug on monthly pageviews
[18:26:45] <joal>	 Thanks fdans for caring that!
[18:26:51] <mforns>	 :]
[18:26:55] <AndyRussG>	 Though in that case any bucket would be either all consistent or non-consistent, since consistency would be a dimension
[18:27:06] <mforns>	 AndyRussG, yes, right
[18:27:31] <AndyRussG>	 But there must be some way to make it calculate the ration when you're not splitting on that dimension
[18:28:25] <joal>	 AndyRussG: Pivot doesn't let you do it, but Druid (the backend computation system), definitely does those things (compute stuff from measures on the fly_)
[18:28:45] <AndyRussG>	 hmmm
[18:29:28] <mforns>	 AndyRussG, even if that is not a measure, but just a dimension, pivot will be able to show the split by consistency and you'll be able to compare, at least visually
[18:30:27] <AndyRussG>	 Another fun thing might be to split on custom aggregations (for example, to get a useful view of geo distribution, compare requests in a single large region vs. total requests in a bunch of smaller regions)
[18:30:47] <AndyRussG>	 Ah hmm interesting
[18:34:13] <AndyRussG>	 Mmmm I was also thinking about the privacy thing to allow some data to be public. Maybe, instead of making a scrubbed version of Druid content, placing a scrubbing procedure between Druid and the UI. That way you wouldn't get weird data in Druid, but the UI could say, for example, "< 50 / hour" or something (for example, for regions where a small number of results for a very unusual bucket could
[18:34:15] <AndyRussG>	 be potentially privacy-compromising for someone)
[18:34:27] <AndyRussG>	 and that way you'd still get correct aggregate numbers
[18:35:05] <nuria>	 lzia: we paused the ticket while waiting for feedback meaning that when feedback from research comes we likely have work to do (re: https://phabricator.wikimedia.org/T153923)
[18:35:32] <wikibugs>	 06Analytics-Kanban, 06Research-and-Data: Coordinate with research to vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#2982761 (10Nuria) Moving to "paused" as we wait for feedback from research.
[18:37:14] <wikibugs>	 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2982767 (10Nuria) > The application that creates pageview dumps should escape or filter newlines (and tabs and other unexpected cha...
[18:37:56] <wikibugs>	 10Analytics: Utility that creates pageview dumps should escape new lines - https://phabricator.wikimedia.org/T156656#2982769 (10Nuria)
[18:38:23] <wikibugs>	 10Analytics, 10Analytics-Dashiki: dashiki should execute tests on jenkins - https://phabricator.wikimedia.org/T156657#2982784 (10Nuria)
[18:41:01] <mforns>	 AndyRussG, yes, we considered a scrubbing procedure between Druid and the UI, among other options. It wouldn't be trivial though, given that open-source pivot is not maintained any more...
[18:41:58] <mforns>	 This quarter we are not prioritizing anonymization, but have plans to be looking into anonymization this year 2017 for sure
[18:43:43] <nuria>	 a-team: about to deploy latest change to EL 
[18:43:50] <mforns>	 k
[18:46:26] <AndyRussG>	 mforns: cool! Yeah it's not been prioritized for us, either.... so it goes... hopefully we'll also get to work on it sometime... /me returns to duly prioritized but less exciting work ;p
[18:49:01] <nuria>	 looks like i am going to have to wait for otto as pip is not installed 
[18:49:05] <nuria>	 https://www.irccloud.com/pastebin/ObIPNhmS/
[18:50:41] <icinga-wm>	 PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: processor/client-side-01
[18:51:53] <wikibugs>	 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2982826 (10Tgr) >>! In T144100#2982767, @Nuria wrote: > But it is not only that what needs doing right? faulty requests that are re...
[18:56:48] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning.
[18:57:21] <wikibugs>	 (03PS1) 10Joal: Update oozie pageview loading into druid jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/335067
[18:57:28] <joal>	 mforns: if you have minute --^
[18:58:13] <joal>	 a-team, leaving for today - tomorrow folks!
[18:58:23] <mforns>	 bye joal ! will look into the patch
[18:58:52] <fdans>	 see you tomorrow a-team!
[18:59:00] <mforns>	 bye fdans !
[19:02:24] <wikibugs>	 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#2982894 (10Sadads) @Nuria the challenge has been that @legoktm did a good start, but capacity...
[19:05:21] <wikibugs>	 (03CR) 10Mforns: [C: 031] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/335067 (owner: 10Joal)
[19:06:21] <wikibugs>	 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Wikimedia-General-or-Unknown, 13Patch-For-Review: Implement Schema:ExternalLinksChange - https://phabricator.wikimedia.org/T115119#2982898 (10Nuria) @Sadads:Sorry i cannot be of more help but analytics team does not instrumen...
[19:07:55] <wikibugs>	 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2982902 (10Nuria)
[19:08:07] <wikibugs>	 10Analytics, 10Dumps-Generation, 05Security: Pageview dumps incorrectly formatted, looks like a result of possibly malicious activity - https://phabricator.wikimedia.org/T144100#2588550 (10Nuria) Indeed. Added subtask.
[19:10:12] <wikibugs>	 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, and 2 others: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#2982912 (10DarTar) a:05DarTar>03ellery
[19:42:02] <nuria>	 milimetric: I can take care of piwik of financial report, i want to make sure it is not below 15.wikimedia.org cause in that case it might make sense to reuse same site id
[19:42:15] <nuria>	 as pageview url woul differentiate both reports
[19:43:01] <milimetric>	 sure, up to you nuria
[19:51:21] <wikibugs>	 06Analytics-Kanban: Pageview Jobs: Make workflows easier to maintain using a variable instead of repeating some complex value accross the files - https://phabricator.wikimedia.org/T156668#2983116 (10Nuria)
[19:51:38] <wikibugs>	 (03CR) 10Nuria: [V: 032 C: 032] "Looks good, would be good to have a ticket to keep track of what is being deployed to cluster . Created task: https://phabricator.wikimedi" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/335067 (owner: 10Joal)
[19:53:42] <wikibugs>	 (03PS2) 10Nuria: Update oozie pageview loading into druid jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/335067 (https://phabricator.wikimedia.org/T156668) (owner: 10Joal)
[19:54:45] <wikibugs>	 (03CR) 10Nuria: [V: 032 C: 032] Update oozie pageview loading into druid jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/335067 (https://phabricator.wikimedia.org/T156668) (owner: 10Joal)
[19:58:32] <wikibugs>	 06Analytics-Kanban, 06Research-and-Data: Coordinate with research to vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#2895122 (10DarTar) @Nuria let's go over this at our next 1:1 since it goes beyond Erik's involvement (wikistats transition) and we need to scope it, it's un...
[20:17:56] <wikibugs>	 10Analytics, 06Developer-Relations, 10MediaWiki-API, 06Reading-Admin, and 2 others: Is User-Agent data PII when associated with Action API requests? - https://phabricator.wikimedia.org/T154912#2983267 (10Nuria)
[20:23:27] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Add user_agent_map field to EventCapsule - https://phabricator.wikimedia.org/T153207#2983311 (10Tbayer) Please note that this task is about //adding// the user agent map to the capsule and storing it alongside the existing raw user agent, not...
[20:33:19] <wikibugs>	 10Analytics-Tech-community-metrics: Mismatch between six names and certain email address in mediawiki-identities data - https://phabricator.wikimedia.org/T123643#2983360 (10Aklapper) a:05Dicortazar>03None
[20:33:44] <wikibugs>	 10Analytics-Tech-community-metrics: korma: Empty "subject" and "creator" fields for mailing list thread on mls.html - https://phabricator.wikimedia.org/T116284#2983363 (10Aklapper) a:05Dicortazar>03None Unassigning until this shows up again in a Kibana world.
[20:35:36] <wikibugs>	 10Analytics-Tech-community-metrics: korma: demographics.html: "Tickets participants" has "184 attracted" data for 1year<x≤0.5years though Bugzilla closed in 11/2014 - https://phabricator.wikimedia.org/T120569#2983383 (10Aklapper)
[20:35:44] <wikibugs>	 10Analytics-Tech-community-metrics: korma: top-contributors.html displays comma as "Location" when a person has more than one affiliation - https://phabricator.wikimedia.org/T123926#2983385 (10Aklapper)
[20:36:04] <wikibugs>	 10Analytics-Tech-community-metrics: korma: GrimoireLib sometimes displays different names for same user ID; link does not display (existing) contributor data - https://phabricator.wikimedia.org/T140299#2983386 (10Aklapper)
[20:40:18] <wikibugs>	 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, 10Research-management: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#2983412 (10RobH) Please note that since this is no longer an active hardware request, I'm going to remove the #project so we don't get used...
[20:45:14] <wikibugs>	 10Analytics-Tech-community-metrics: Provide equivalent of "SCR: People uploading patchsets vs. Reviewers per month" in Kibana - https://phabricator.wikimedia.org/T151559#2983441 (10Aklapper) p:05Low>03Normal
[20:45:17] <wikibugs>	 10Analytics-Tech-community-metrics: Provide equivalent of "SCR: Code review users vs. Code review committers" in Kibana - https://phabricator.wikimedia.org/T151558#2983443 (10Aklapper) p:05Low>03Normal
[20:45:27] <wikibugs>	 10Analytics-Tech-community-metrics: Missing time units for percentile values - https://phabricator.wikimedia.org/T145425#2983445 (10Aklapper) p:05Normal>03Low
[20:51:45] <wikibugs>	 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 13Patch-For-Review, and 2 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#2983476 (10Jdlrobson) Before we continue bumping this to 24hrs I'd like to suggest we address T156666 and...
[21:00:25] <wikibugs>	 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 13Patch-For-Review, and 2 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#2983492 (10mobrovac) >>! In T156411#2983476, @Jdlrobson wrote: > Before we continue bumping this to 24hrs...
[21:10:23] <wikibugs>	 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 13Patch-For-Review, and 2 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#2983513 (10Jdlrobson) @mobrovac sure. Previously we only replayed only one hour of edits. The benefit of d...
[21:11:16] <wikibugs>	 10Analytics, 10DBA: Json_extract available on analytics-store.eqiad.wmnet - https://phabricator.wikimedia.org/T156681#2983530 (10Nuria)
[21:11:43] <wikibugs>	 10Analytics, 06Developer-Relations, 10MediaWiki-API, 06Reading-Admin, and 2 others: Is User-Agent data PII when associated with Action API requests? - https://phabricator.wikimedia.org/T154912#2983532 (10bd808) >>! In T154912#2983231, @Nuria wrote: > Let's plan on deleting all PII data and if/when someone...
[21:14:14] <wikibugs>	 10Analytics, 06Developer-Relations, 10MediaWiki-API, 06Reading-Admin, and 2 others: Is User-Agent data PII when associated with Action API requests? - https://phabricator.wikimedia.org/T154912#2983537 (10bd808) >>! In T154912#2983231, @Nuria wrote: > @bd808 and @tgr: in order for purging to occur recurrent...
[21:14:23] <wikibugs>	 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Add user_agent_map field to EventCapsule - https://phabricator.wikimedia.org/T153207#2983538 (10Nuria) >Please note that this task is about adding the user agent map to the capsule and storing it alongside the existing raw user agent, not abou...
[21:14:35] <wikibugs>	 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 13Patch-For-Review, and 2 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#2983539 (10Jdlrobson) (I should add that we do decay scores over time using a half life of 1.5hrs so this...
[21:22:13] <wikibugs>	 10Analytics-Dashiki, 06Analytics-Kanban, 13Patch-For-Review: Add extension and category  (ala Eventlogging) for DashikiConfigs - https://phabricator.wikimedia.org/T125403#2983563 (10Milimetric) Currently in stage https://www.mediawiki.org/wiki/Review_queue#Preparing_for_deployment where I have to first get c...
[21:57:24] <wikibugs>	 (03PS1) 10EBernhardson: Script to drop mediawiki log partitions in HDFS [analytics/refinery] - 10https://gerrit.wikimedia.org/r/335158
[22:02:00] <wikibugs>	 10Analytics, 06Developer-Relations, 10MediaWiki-API, 06Reading-Admin, and 2 others: Is User-Agent data PII when associated with Action API requests? - https://phabricator.wikimedia.org/T154912#2983680 (10EBernhardson) >>! In T154912#2983537, @bd808 wrote: >>>! In T154912#2983231, @Nuria wrote: >> @bd808 an...
[23:40:23] <wikibugs>	 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 13Patch-For-Review, and 2 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#2984028 (10mobrovac) I see, thank you @Jdlrobson for the explanation. It sounds like we need considerable...