[01:17:59] 10Quarry: Redirect Toolforge Quarry page to Cloud VPS Quarry - https://phabricator.wikimedia.org/T175881#3606587 (10Quiddity) [01:54:43] 10Quarry: Redirect Toolforge Quarry page to Cloud VPS Quarry - https://phabricator.wikimedia.org/T175881#3606587 (10zhuyifei1999) From [[https://gerrit.wikimedia.org/r/#/c/304764/|Gerrit change 304764]], I *think* this is Yuvi's attempt to move Quarry to Toolforge. There are quite some benefits, such as better r... [01:59:15] 10Quarry, 10Toolforge: Redirect Toolforge Quarry page to Cloud VPS Quarry - https://phabricator.wikimedia.org/T175881#3606633 (10Quiddity) [02:26:54] 10Quarry, 10Toolforge: Redirect Toolforge Quarry page to Cloud VPS Quarry - https://phabricator.wikimedia.org/T175881#3606639 (10MZMcBride) At minimum, putting a big note at the top of seems reasonable. Better would be an HTTP 301 or 302 to . [11:05:06] * elukey lunch! [12:00:39] taking a break a-eam [12:31:49] need to step away for ~1h due to unexpected errands folks [12:31:55] * elukey afk but reachable [12:38:11] will be at the doctor's until around 18:00 UTC... :( [12:45:38] milimetric _hug_ [13:06:09] hellooooo [13:15:48] elukey: , i might have a working prometheus deb package shortly... [13:16:11] prometheus jmx exporter* [13:16:46] ottomata: I am testing the deploy in labs, why do we need a deb? [13:17:25] also I think that the project on gh releases a deb automatically [13:17:44] cause scap sucks for this [13:17:48] dont' want to maintain list of hsots [13:18:03] oh ya? [13:18:22] sure but we need to maintain a deb rather than relying on the scap repo [13:18:34] (that is done for other things like cassandra, etc..) [13:18:41] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Select candidate jobs for transferring to the new infrastucture - https://phabricator.wikimedia.org/T175210#3607836 (10mobrovac) 05Resolved>03Open Sure. [13:22:26] yeahhHHHHHh i dunnooooo i think that is way better, its how everything else is done, cassandra is just doing it because it was easier [13:22:34] talked to eric e yesterday and he thought a deb was a good idea [13:23:01] sure, my point is: we all need to agree with 1 single way of doing this otherwise we'll get crazy in no time :D [13:24:04] ya, for sure, if there's a deb i'm sure they won't mind using it [13:24:27] but also, elukey, i tlaked ot eric, he said they use icinga/graphite for alerting [13:24:32] soooo we might need to do jmx trans anyway? [13:24:43] unless we want to pioneer alerts too [13:24:50] did you talk to godog about that? [13:25:00] nope didn't manage to [13:32:48] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: EventLogging subscriber in ready state but not sending tracked events - https://phabricator.wikimedia.org/T175918#3607899 (10phuedx) [13:33:05] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: EventLogging subscriber module in ready state but not sending tracked events - https://phabricator.wikimedia.org/T175918#3607901 (10phuedx) [13:40:54] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, 10Services (designing): Requests for new JobQueue monitoring capabilities - https://phabricator.wikimedia.org/T175780#3607928 (10mobrovac) [13:52:36] mforns: holaaa [13:52:46] elukey, hello! [13:52:46] I have a new feature to add to eventlogging_cleaner [13:52:50] aha [13:53:28] this came up after a chat with Riccardo [13:53:55] is it the solution to all our problems? [13:54:04] :] [13:54:15] if we put the script in cron now to purge say the last couple of days, then there might be the case in which $something fails and for some reason we don't receive the alert [13:54:20] :D [13:54:29] email goes to spam, we are all busy, etc... [13:54:38] aha [13:55:04] if we don't act quickly then we risk to have "holes" of sanitization [13:55:07] BUT [13:55:19] we could have a special option called "start-ts-file" [13:55:24] containing the start ts [13:55:31] the script grabs it, and start from it [13:55:41] ignoring --newer-than [13:55:56] when it completes all the batches, it puts end_ts in the file [13:56:07] so it will be picked up during the next run [13:56:14] gotcha [13:56:18] does it make sense? [13:56:30] we could also commit to the db [13:56:33] as we want [13:57:02] yes, we just should make sure that this file is created with the right permits, so that executions by both cron and us can read it [13:57:23] yes, db makes more sense maybe? [13:59:19] no preference [13:59:57] maybe a file is simpler [14:00:09] yes for sure [14:49:42] milimetric: would you have minute for me? [14:50:11] Arf no, forgot he's o the doctor :( [14:50:27] mforns maybe? [14:50:41] joal hi! batcave? [14:50:45] OMW ! [14:53:51] 10Analytics-Cluster, 10Analytics-Kanban: Use Prometheus for Kafka JMX metrics instead of jmxtrans - https://phabricator.wikimedia.org/T175922#3608057 (10Ottomata) [14:54:44] ottomata: I already had a task--^ [14:55:23] https://phabricator.wikimedia.org/T175344 [14:55:34] maybe more generic [14:55:49] let's use yours for jumbo [14:55:53] oh you did! [14:55:53] 10Analytics-Cluster, 10Analytics-Kanban: Port Kafka alerts from check_graphite to check_prometheus - https://phabricator.wikimedia.org/T175923#3608073 (10Ottomata) [14:55:59] oh ooops [14:56:31] makes more sense to have a single one for kafka jumbo [14:56:36] and one generic for hadoop etc.. [14:56:39] no problem :) [14:56:49] ok...should we merge yours into mine? [14:57:27] I'd prefer to keep it open to track Q2's work for Hadoop [14:57:38] I'll rename it maybe [14:59:05] ok [15:00:18] ping fdans ottomata [15:15:22] ottomata: shall I assume usr/share/java/prometheus for the new pkg? [15:16:24] elukey: ya if folks +1 that stuff [15:17:00] sure sure, but I can amend the code review in the meantime :) [15:19:01] ottomata: ah so something like /usr/share/java/prometheus/jmx_prometheus_httpserver-$(DEB_VERSION_UPSTREAM).jar.. because I'd need to specify DEB_VERSION_UPSTREAM in puppet [15:19:12] (the javaagent needs the full path of the jar) [15:19:59] elukey: it creates a symlink there too [15:19:59] so [15:20:04] you can leave off the version [15:20:45] /usr/share/java/prometheus/jmx_prometheus_httpserver.jar [15:20:48] wonderful [15:25:58] 10Analytics-Kanban, 10Beta-Cluster-Infrastructure, 10Wikimedia-Stream, 10Patch-For-Review: Decom RCStream in Beta Cluster - https://phabricator.wikimedia.org/T172356#3608191 (10Ottomata) Just made a patch to use EventBus for RCFeed instead of RCStream. If we merge that, we can remove the RCStream puppet m... [15:29:23] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: EventLogging subscriber module in ready state but not sending tracked events - https://phabricator.wikimedia.org/T175918#3608213 (10phuedx) [15:32:05] ping elukey [15:32:48] comingg [15:46:47] 10Analytics, 10User-Elukey: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344#3590888 (10Nuria) Cassandra, zookeeper, druid, hadoop, kafka [16:01:55] 10Analytics, 10Analytics-EventLogging, 10Page-Previews, 10Readers-Web-Backlog: EventLogging subscriber module in ready state but not sending tracked events - https://phabricator.wikimedia.org/T175918#3608298 (10phuedx) [16:03:36] 10Analytics-Kanban, 10Reading-analysis: Final Vetting of Family Wide unique devices data - https://phabricator.wikimedia.org/T169550#3608303 (10Nuria) Ping @Tbayer what is the status of this vetting? [16:03:59] 10Analytics-Kanban: Collaborate with zero on asiacell report - https://phabricator.wikimedia.org/T161326#3608304 (10Nuria) 05Open>03Resolved [16:04:56] 10Analytics, 10Analytics-Dashiki, 10Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3608305 (10Nuria) [16:10:51] 10Analytics-Kanban: Add monthly unique devices dataset to pivot - https://phabricator.wikimedia.org/T163327#3193688 (10Nuria) We cannot update pivot due to it being closed source so we will not be able to add dataset to pivot (thus far it cannot deal with monthly granularity), it can however be added to druid an... [16:11:01] 10Analytics-Kanban: Add monthly unique devices dataset to Druid - https://phabricator.wikimedia.org/T163327#3608312 (10Nuria) [16:11:56] 10Analytics, 10MediaWiki-extensions-WikimediaEvents, 10The-Wikipedia-Library, 10Patch-For-Review: ExternalLinksChange Logging instrumentation is completely broken - https://phabricator.wikimedia.org/T162365#3608314 (10Nuria) [16:13:29] 10Analytics-Kanban: vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#3608317 (10Nuria) Will set up meeting [16:16:12] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Replacement of stat1002 and stat1003 - https://phabricator.wikimedia.org/T152712#3608318 (10Nuria) [16:17:49] 10Analytics-Kanban: Chose how to deal with "Infinity" value for Banners - https://phabricator.wikimedia.org/T175248#3587723 (10Nuria) ping @Jseddon Can we filter the incorrect rows? [16:21:13] 10Analytics-Kanban, 10Analytics-Wikistats: Backend for wikistats 2.0 - https://phabricator.wikimedia.org/T156384#3608329 (10Nuria) [16:21:15] 10Analytics-Kanban: Design document for wikistats prototype backend - https://phabricator.wikimedia.org/T162817#3608328 (10Nuria) 05Open>03Resolved [16:21:41] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608331 (10Pchelolo) After the patch was deployed the situation improved a lot, but we've got a 5 Mb event today: https://people.wikimedia.org/~p... [16:21:44] 10Analytics-Kanban: Final steps to expose project family unique devices data - https://phabricator.wikimedia.org/T167539#3608332 (10Nuria) [16:22:12] 10Analytics-Kanban: Final steps to expose project family unique devices data - https://phabricator.wikimedia.org/T167539#3336133 (10Nuria) [16:22:14] 10Analytics-Kanban, 10Reading-analysis: Final Vetting of Family Wide unique devices data - https://phabricator.wikimedia.org/T169550#3608333 (10Nuria) [16:25:48] 10Analytics-Kanban, 10Analytics-Wikistats: Re-read Round 2 feedback on wikistats on mediawiki and make any critical items into tasks - https://phabricator.wikimedia.org/T167674#3608353 (10Nuria) [16:26:52] 10Analytics-Kanban, 10Easy: Investigate requests flagged as pageview in analytics header coming from bots - https://phabricator.wikimedia.org/T135251#3608354 (10Nuria) a:03Nuria [16:30:16] 10Analytics, 10EventBus, 10Easy, 10Services (watching): EventBus logs don't show up in logstash - https://phabricator.wikimedia.org/T153029#3608356 (10Nuria) [16:32:57] 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats 2.0 UI second deployment/iteration - https://phabricator.wikimedia.org/T170460#3608363 (10Nuria) [16:32:59] 10Analytics-Kanban, 10Analytics-Wikistats: Addition of (mock) Active Editors metric - https://phabricator.wikimedia.org/T170463#3608362 (10Nuria) 05Open>03declined [16:33:07] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608364 (10daniel) @Pchelolo disabling escaping of non-ascii characters would probably reduce the size to a fourth... I'm not sure what kind of... [16:33:52] 10Analytics, 10Analytics-Cluster: CamusPartitionChecker does not work when topic names have '.' or '-' in them. - https://phabricator.wikimedia.org/T171099#3608367 (10Nuria) [16:34:31] 10Analytics-Cluster, 10Analytics-Kanban, 10Language-Team, 10MediaWiki-extensions-UniversalLanguageSelector, and 3 others: Migrate table creation query to oozie for interlanguage links - https://phabricator.wikimedia.org/T170764#3608368 (10Nuria) [16:36:41] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3590071 (10mobrovac) >>! In T175316#3608364, @daniel wrote: > We can tweak the chunk size - more jobs, or larger jobs, your pick. Since in the n... [16:39:57] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608381 (10daniel) @Pchelolo actually, can you confirm how many entries there were in the "pages" parameter? With the latest patches deployed, th... [16:41:44] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608386 (10Pchelolo) > @Pchelolo actually, can you confirm how many entries there were in the "pages" parameter? With the latest patches deployed... [16:43:03] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608388 (10daniel) @mobrovac how about a very large number of very small jobs? e.g. a million jobs to purge a million pages from cdn? [16:47:45] * elukey off!! [16:56:06] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608408 (10daniel) > For now, let's just try to get the size of the jobs below the 4MB mark? :) If you fix your encoding ;) [17:25:13] 10Analytics-Kanban, 10Analytics-Wikistats: Use daily granularity for 1-month time ranges - https://phabricator.wikimedia.org/T173372#3608558 (10fdans) //The X axis shrinks in 1-month and 3-month mode (compared to the 1-year mode). This should always be the same size regardless of the data, and that should be a... [17:32:50] 10Analytics-Tech-community-metrics, 10Gerrit, 10Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3608614 (10Paladox) I wonder if we can try https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#fix-change ? [17:34:13] 10Analytics-Tech-community-metrics, 10Gerrit, 10Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3608618 (10Paladox) Ah, running https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#check-change results in "pr... [17:35:51] !log restaring eventlogging processor(s) with MySQL blacklist of PageCreation schema [17:35:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:04:52] (03CR) 10Nuria: [V: 031 C: 031] "Pending @elukey confirmation I think this is ready to be merged and so it is companios change https://gerrit.wikimedia.org/r/#/c/376640/" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355601 (https://phabricator.wikimedia.org/T162034) (owner: 10Mforns) [18:11:55] 10Analytics: Screen cast for how to use Pivot (5 minutes) - https://phabricator.wikimedia.org/T148776#3608770 (10Nuria) 05Open>03declined [18:15:08] 10Analytics-Kanban: https://dumps.wikimedia.org/other/pageviews/ needs a README - https://phabricator.wikimedia.org/T167033#3608773 (10Nuria) a:03Milimetric [19:14:17] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608922 (10GWicke) Looks like adding the `JSON_UNESCAPED_UNICODE` flag should do it: http://php.net/manual/en/function.json-encode.php [19:18:40] 10Analytics, 10EventBus, 10Wikidata, 10Patch-For-Review, 10Services (designing): Very large jobs posted by Wikidata - https://phabricator.wikimedia.org/T175316#3608928 (10Pchelolo) >>! In T175316#3608922, @GWicke wrote: > Looks like adding the `JSON_UNESCAPED_UNICODE` flag should do it: http://php.net/ma... [19:18:56] milimetric: you around? [19:34:33] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, 10Services (designing): Split ChangeProp metrics by wiki - https://phabricator.wikimedia.org/T175952#3608969 (10Pchelolo) [19:36:40] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, 10Services (designing): Split ChangeProp metrics by wiki - https://phabricator.wikimedia.org/T175952#3608969 (10Ottomata) This might be something worth sending into Druid (& Pivot), if the point is more about exploration and finding problems as... [19:37:49] nuria_: , yt? [19:38:04] yessir looking at $$ for travels [19:38:28] quick outline check of what i'm thinking for this talk: [19:38:41] • Describe mediawiki, webrequest and eventlogging datasources, their scale, interesting facts around them, and how briefly how we get them into Hadoop. [19:38:41] • Describe how these datasources are then ingested into Druid, realtime or not. [19:38:41] • Demo pivot?  Superset? [19:38:42] • Talk about future work, wikistats, AQS with druid, etc.? [19:38:57] sharing very wip doc with you [19:39:42] ottomata: ok, i will start with "framing scale" [19:40:00] framing scale? [19:40:11] ottomata: teh scale at which we operate [19:40:13] *the [19:40:24] such us requests per sec [19:40:27] oh ya, got a lot of that from an old talk, am updating with stuf [19:40:29] see doc if you like [19:40:37] k looking [20:08:46] DarTar: Hi [20:08:57] hey joal [20:09:21] DarTar: Long time no see :) How are you? [20:09:38] not bad [20:10:06] sorry I’ve been missing pretty much all the chillout sessions, joal [20:10:21] DarTar: Quick one, do you have a contact address for Florian Lemmerich? I think he's a research fellow [20:10:34] yes [20:10:40] -> DM [20:10:58] DarTar: I'd like to spend a few minutes with him to discuss cluster best practives [20:10:58] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Options for implementing JobQueue statistics methods - https://phabricator.wikimedia.org/T175957#3609097 (10Pchelolo) [20:11:28] joal: makes sense [20:13:17] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 2 others: Add unit tests to EventBus extension - https://phabricator.wikimedia.org/T175958#3609114 (10Pchelolo) [20:52:02] 10Analytics-Tech-community-metrics: Empty or incorrect data on Kibana's "Git-Demographics" dashboard - https://phabricator.wikimedia.org/T171240#3609245 (10Aklapper) [21:05:21] 10Analytics-Kanban: vet metrics calculated from the data lake - https://phabricator.wikimedia.org/T153923#3609290 (10Erik_Zachte) Please do. Tomorrow any time till 12 AM PDT works for me. Preferably a bit earlier. [21:20:06] Leaving for tonight a-team [21:20:11] byeeee joal [21:20:12] :] [21:57:37] 10Analytics, 10Phabricator: Create phabricator space for tickets with legal restrictions - https://phabricator.wikimedia.org/T174675#3569915 (10ZhouZ) Thanks also Legal might also want to create a separate Legal space in addition to the one with Analytics folks [23:59:54] (03PS3) 10Dzahn: Update svn.wikimedia.org links to phabricator [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/316289 (https://phabricator.wikimedia.org/T64570) (owner: 10Paladox) [23:59:59] (03CR) 10Dzahn: [C: 032] Update svn.wikimedia.org links to phabricator [analytics/wikistats] - 10https://gerrit.wikimedia.org/r/316289 (https://phabricator.wikimedia.org/T64570) (owner: 10Paladox)