[06:58:45] 10Analytics, 10Analytics-Wikistats: Correct order of metric types on Dashboard navbar/menu - https://phabricator.wikimedia.org/T198168#4314171 (10sahil505) [06:59:49] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Correct order of metric types on Dashboard navbar/menu - https://phabricator.wikimedia.org/T198168#4314171 (10sahil505) p:05Triage>03Normal [07:07:50] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4294420 (10mmodell) I also deactivated {rOPVK} [07:08:09] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4314201 (10mmodell) [07:12:23] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4314204 (10mmodell) Github now has an archive feature, do we still delete them rather than just archive on github? [07:17:36] (03PS1) 10Sahil505: Corrected order of metric types on Dashboard navbar [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442022 (https://phabricator.wikimedia.org/T198168) [07:26:53] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4314230 (10elukey) [07:45:03] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4314285 (10elukey) https://phabricator.wikimedia.org/diffusion/OPKT/ deactivated [07:45:19] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4314288 (10elukey) [07:45:45] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097#4314289 (10elukey) https://phabricator.wikimedia.org/diffusion/OPJM/ deactivated [07:45:55] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097#4314290 (10elukey) [07:49:41] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097#4311727 (10mmodell) archived the github mirror [07:51:27] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4311739 (10mmodell) archived https://github.com/wikimedia/operations-puppet-kafkatee [08:04:28] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4314338 (10elukey) [08:04:37] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4314342 (10elukey) [08:04:40] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503#4294420 (10elukey) 05Open>03Resolved [08:05:07] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4005170 (10elukey) [08:05:12] 10Analytics, 10Cleanup, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097#4314343 (10elukey) 05Open>03Resolved [08:05:36] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4314350 (10elukey) [08:09:37] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4314368 (10elukey) [08:09:42] 10Analytics, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Archive operations/puppet/kafkatee repository - https://phabricator.wikimedia.org/T198098#4314367 (10elukey) 05Open>03Resolved [08:14:43] as all this spam suggests, submodules merged :) [08:47:12] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 3 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4314452 (10Pchelolo) [08:47:18] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Migrate CirrusSearch jobs to Kafka queue - https://phabricator.wikimedia.org/T189137#4314450 (10Pchelolo) 05Open>03Resolved [09:05:56] 10Analytics, 10Operations, 10procurement, 10User-Elukey: eqiad (3): hardware refresh for dbstore1002 - https://phabricator.wikimedia.org/T198174#4314468 (10elukey) [09:22:48] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 3 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4314574 (10mobrovac) [09:22:52] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#4314575 (10mobrovac) [09:22:54] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 4 others: Create custom per-job metric reporters capability - https://phabricator.wikimedia.org/T182274#4314573 (10mobrovac) [09:48:22] 10Analytics, 10Analytics-Wikistats: Disable filter & split radio switches for metrics & reduce the height of wiki div in sidebar of detail page - https://phabricator.wikimedia.org/T198183#4314636 (10sahil505) [09:49:16] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Disable filter & split radio switches for metrics & reduce the height of wiki div in sidebar of detail page - https://phabricator.wikimedia.org/T198183#4314636 (10sahil505) p:05Triage>03Normal [09:54:04] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: Display of radio buttons in Wikistats 2 is somewhat confusing - https://phabricator.wikimedia.org/T183185#4314661 (10sahil505) @Amitjoki : I was wondering if you will be submitting a patch for new Semantic's radio style button, if not I can merge this w... [10:01:24] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Varnishkafka does not play well with varnish 5.2 - https://phabricator.wikimedia.org/T177647#4314685 (10elukey) Tested with varnish 6.0 + https://docs.confluent.io/current/quickstart/cos-quickstart.html on a Vagrant VM, everything seems ok after multiple stress... [10:01:29] brb! [10:46:21] (03CR) 10Addshore: [C: 04-1] Track new API reported maxlag for wikidata on grafana dashboard (031 comment) [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (https://phabricator.wikimedia.org/T196868) (owner: 10Jonas Kress (WMDE)) [10:54:08] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Varnishkafka does not play well with varnish 5.2 - https://phabricator.wikimedia.org/T177647#4314832 (10elukey) Created the new varnishv51 branch and added some info in the README about the new branch policy. I think that everything is don... [10:57:34] aaand varnishkafka ready for 5.2/6.0 --^ [11:18:31] * elukey misses joal [11:51:45] (03PS1) 10Mforns: Add inline comments to WhitelistSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/442076 (https://phabricator.wikimedia.org/T195269) [12:27:39] just tested piwik 3.2.1 in labs (version before the name change to matomo) [12:27:45] and it seems working fine [12:27:54] but we'll have to take an outage to upgrade the db [12:38:35] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Update piwik to latest stable - https://phabricator.wikimedia.org/T192298#4315242 (10elukey) p:05Low>03Normal [12:41:44] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Update piwik to latest stable - https://phabricator.wikimedia.org/T192298#4315266 (10elukey) I tested today in labs an upgrade from the current version 2.17 to 3.2.1, the latest stable before the rebrand to matomo. Everything worked fine, but the update functio... [13:07:47] * elukey afk for a bit (replace lunch with errands :) [13:16:45] (03CR) 10Ottomata: [C: 032] Add inline comments to WhitelistSanitization [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/442076 (https://phabricator.wikimedia.org/T195269) (owner: 10Mforns) [13:43:48] elukey: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/440507/ [13:43:52] also o/ [13:43:52] :) [14:01:26] ottomata: one nit and then +2! Thanks :) [14:04:15] OO good catch thanks elukey [14:08:59] Hi elukey - I'm still in the airport with limited connection :) [14:10:33] :) [14:13:06] elukey: I've seen you've been doing some cleaning on gerrit :) [14:15:00] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move piwik's database to db1108 (eventlogging slave) - https://phabricator.wikimedia.org/T198217#4315603 (10elukey) p:05Triage>03Normal [14:17:08] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move piwik's database to db1108 (eventlogging slave) - https://phabricator.wikimedia.org/T198217#4315617 (10elukey) The main question remains security, since as far as I can see Piwik does not support TLS encryption/authentication to the database. The connectio... [14:17:21] joal: oh how did notebook score stuff go? [14:17:25] joal: cleaning day! :) [14:22:30] ottomata: I have a request for you to try making spark work [14:22:42] oh? [14:22:57] ottomata: I didn't make progress on "woah effect" though :( [14:23:09] ottomata: I have the code at home, will ask later when back [14:23:17] And will spend time on it with you after [14:23:18] ayyyye, ok, well even if running on local on small data [14:23:19] its fine [14:23:22] ok [14:23:36] its more about demoing the coolness of the data, and showing how it would be easier if the data was better formatted [14:24:30] ottomata: I have seen some tickets move letting me think that Aaron was taking charge - is that corect? [14:24:44] he was fixing a little discrepency in one of their things [14:24:58] and he commented that he thought the problem shoudl be fixed downstream [14:25:02] sooo eee dunno [14:25:03] we will see [14:25:12] i'm talking to him on thursday for event platform stuff [14:25:15] and some other ores folks before [14:25:49] elukey: fyi just enabled eqiad -> codfw job topic mirroring [14:25:59] o/ I'm willing to discuss JSON formats. [14:26:00] we ight see MirrorMaker alerts because of that cirrus search job [14:26:00] not sure [14:26:09] halfak: hi! ya we are trying to put together an example [14:26:12] But taking away "true" and "false" from our predictions seems wrong :P [14:26:47] elu key: if the cirrus job causes problems [14:26:50] i'll just blacklist that one [14:27:43] halfak: true and false are minor, and might be ok, not sure. it def would be easier if they weren't keywords, but i thikn those might be ok. others are harder, like having '.' in the column names [14:28:03] and also keying by rev id...but that one will be harder i think to figure out for yall, not sure [14:28:29] ottomata, seems like you could just use an escape/normalization strategy for "." [14:28:58] yes but where? we want integration to be easy for more than just hive [14:29:08] ottomata: ack! [14:29:29] to do that the events need to be normalized earlier. we are doing that a little bit in change prop right now, (getting rid of the rev_id keys), but petr doesn't want to start doing more and more stuff there, and i tend to agree with him [14:30:16] we're trying to standardize event schemas, and it'd be really helpful if the score events weren't a special case of change event [14:30:41] halfak: i wonder if we could make ores itself emit the events in a compatible format? instead of relying on somethign else to transform them? [14:31:01] it would just http post to eventbus after the scores have been calculated [14:31:01] ottomata, :| That seems so backwards. [14:31:10] which ^ ? [14:31:11] Compatible with what? It's compatible with the tools that use it. [14:31:41] compatible with common data stores...i guess? its compatible now if you are parsing the events directly [14:31:48] with code [14:32:01] I think that, if a consumer needs the data format to change in some what, then the consumer should make the relevant modifications. [14:32:12] *way [14:33:04] perhaps! but in this case the consumer is not necessarily well defined, and we want to make it easier for any consumer (data store integration is mostly what i'm thinking of) to use. [14:34:25] anyway, halfak didn't mean to start this convo now [14:34:44] Gotcha. OK I'll wait and/or put my thoughts where it makes more sense. [14:39:23] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Operations, 10Services (watching): Stop and remove old job runners - https://phabricator.wikimedia.org/T198220#4315678 (10Pchelolo) p:05Triage>03Normal [14:40:02] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Partially purge MobileWikiAppiOSUserHistory eventlogging schema - https://phabricator.wikimedia.org/T195269#4315696 (10mforns) Sorry, this change was not meant to be linked to this task... please ignore. [14:43:32] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 3 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4315710 (10mobrovac) [14:43:36] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, and 3 others: [EPIC] Develop a JobQueue backend based on EventBus - https://phabricator.wikimedia.org/T157088#4315711 (10mobrovac) [14:43:43] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Operations, 10Services (watching): Stop and remove old job runners - https://phabricator.wikimedia.org/T198220#4315709 (10mobrovac) [14:43:48] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Operations, and 2 others: Stop and remove old job runners - https://phabricator.wikimedia.org/T198220#4315678 (10mobrovac) [15:03:14] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move piwik's database to db1108 (eventlogging slave) - https://phabricator.wikimedia.org/T198217#4315603 (10jcrespo) The current state is that DBAs don't support Piwik, and as a consequence, we will not support db1108 and db1107 at all if they are mixed togheth... [15:03:35] ping milimetric ottomata joal [15:09:16] (03PS1) 10Mforns: Add EventLogging whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/442125 (https://phabricator.wikimedia.org/T193176) [15:09:55] (03PS2) 10Mforns: Add EventLogging whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/442125 (https://phabricator.wikimedia.org/T193176) [15:13:03] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move piwik's database to db1108 (eventlogging slave) - https://phabricator.wikimedia.org/T198217#4315784 (10elukey) >>! In T198217#4315768, @jcrespo wrote: > The current state is that DBAs don't support Piwik, and as a consequence, we will not support db1108 an... [15:35:46] 10Analytics, 10Analytics-Cluster: Requesting account expiration extension - https://phabricator.wikimedia.org/T183291#4315863 (10Jdcc-berkman) Update on this: We've finally secured some developer time and are in the process of signing a contract. Can we extend three more months, please? [15:54:06] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move piwik's database to db1108 (eventlogging slave) - https://phabricator.wikimedia.org/T198217#4315944 (10jcrespo) > Makes sense, is there anything that we can do to make Piwik in a good state so that it can be hosted on db1108? No, the issue is we don't sup... [16:01:35] (03PS5) 10Jonas Kress (WMDE): Track new API reported maxlag for wikidata on grafana dashboard [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (https://phabricator.wikimedia.org/T196868) [16:05:39] 10Analytics, 10Analytics-Kanban, 10User-Elukey: Move piwik's database to db1108 (eventlogging slave) - https://phabricator.wikimedia.org/T198217#4315994 (10elukey) >>! In T198217#4315944, @jcrespo wrote: >> Makes sense, is there anything that we can do to make Piwik in a good state so that it can be hosted o... [16:10:28] 10Analytics: Turn off old geowiki jobs - https://phabricator.wikimedia.org/T190059#4316005 (10Nuria) ping @Milimetric did we turned off the old geowiki jobs? [16:13:55] 10Analytics, 10Analytics-Wikistats: Wikistats Beta: How about historic data - https://phabricator.wikimedia.org/T186124#4316025 (10Nuria) p:05Normal>03Lowest [16:14:11] 10Analytics: AQS unique devices api should report offset/underestimate separately - https://phabricator.wikimedia.org/T164201#4316032 (10Nuria) p:05Normal>03Low [16:15:06] 10Analytics, 10Analytics-Wikistats: Wikistats 2.0 Remaining reports. - https://phabricator.wikimedia.org/T186121#4316045 (10Nuria) p:05Normal>03Lowest [16:15:29] 10Analytics: Better redirect handling for pageview API - https://phabricator.wikimedia.org/T121912#4316051 (10Nuria) p:05Normal>03Low [16:16:35] 10Analytics, 10MediaWiki-Releasing: Create dashboard showing MediaWiki tarball download statistics - https://phabricator.wikimedia.org/T119772#4316069 (10Nuria) a:05Nuria>03None [16:17:56] 10Analytics: Add pagecounts by article and top pagecounts to AQS - https://phabricator.wikimedia.org/T173720#4316080 (10Nuria) p:05Low>03Lowest [16:18:48] 10Analytics, 10Analytics-Wikistats: Wikistats 2.0 Remaining reports. - https://phabricator.wikimedia.org/T186121#4316085 (10Nuria) p:05Lowest>03High [16:20:35] 10Analytics: Kerberos prototype running in labs - https://phabricator.wikimedia.org/T198227#4316098 (10Nuria) [16:24:13] (03CR) 10Addshore: [C: 04-1] "It looks like the file is missing the +x bit (which it needs)." [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/441838 (https://phabricator.wikimedia.org/T196868) (owner: 10Jonas Kress (WMDE)) [16:37:33] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Cassandra on AQS to 2.2.6-wmf5 - https://phabricator.wikimedia.org/T197062#4316238 (10elukey) @Eevans for some reason I didn't check what you asked above, I was probably super tired and didn't test this: ``` elukey@aqs1004:~$ cqls... [16:50:46] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Cassandra on AQS to 2.2.6-wmf5 - https://phabricator.wikimedia.org/T197062#4316325 (10Eevans) >>! In T197062#4316238, @elukey wrote: > @Eevans for some reason I didn't check what you asked above, I was probably super tired and didn... [16:53:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Cassandra on AQS to 2.2.6-wmf5 - https://phabricator.wikimedia.org/T197062#4316345 (10elukey) >>! In T197062#4316325, @Eevans wrote: > Are you going to hack that one host locally, or do you want to pursue fixing the package in a wa... [17:01:31] ping mforns [17:01:46] coming! [17:02:10] 10Analytics, 10Cassandra, 10Operations, 10RESTBase-Cassandra, and 3 others: Upgrade prometheus-jmx-exporter on all services using it - https://phabricator.wikimedia.org/T192948#4316415 (10Pnorman) We don't see anything for the maps team to do on this - at the very least, we don't think it needs any resourc... [17:18:15] * elukey off! [17:19:03] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Upgrade Cassandra on AQS to 2.2.6-wmf5 - https://phabricator.wikimedia.org/T197062#4316521 (10Eevans) >>! In T197062#4316345, @elukey wrote: >>>! In T197062#4316325, @Eevans wrote: >> Are you going to hack that one host locally, or do you... [18:30:42] 10Analytics-Kanban, 10Product-Analytics, 10Reading-analysis: Final Vetting of Family Wide unique devices data - https://phabricator.wikimedia.org/T169550#4316847 (10Nuria) @tbayer: can this task be closed? [19:00:29] Hi team - I FINALLY made it home ! [19:37:56] milimetric: are you around? I have a mediawiki_history question for you if you are. [19:38:33] Hi leila - Not sure if I can help, but please shoot :) [19:39:06] joal: I'll add you to a Meet call then. ;) [19:39:40] joal: link sent. [19:58:31] (03CR) 10Mforns: [V: 032 C: 032] "LGTM! Nice fix" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/442022 (https://phabricator.wikimedia.org/T198168) (owner: 10Sahil505) [19:59:13] (03CR) 10Mforns: [V: 032 C: 032] "Good catch! LGTM" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/441917 (https://phabricator.wikimedia.org/T198122) (owner: 10Sahil505) [20:02:35] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech1 - https://phabricator.wikimedia.org/T198256#4317286 (10Ottomata) [20:03:04] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech1 - https://phabricator.wikimedia.org/T198256#4317302 (10Ottomata) [20:03:07] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, 10Services (watching): Modern Event Platform (with EventLogging of the Future (EoF)) - https://phabricator.wikimedia.org/T185233#3910345 (10Ottomata) [20:03:14] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10TechCom-RFC: RFC: Modern Event Platform - Choose Schema Tech - https://phabricator.wikimedia.org/T198256#4317286 (10Ottomata) [20:23:00] 10Analytics: [Wikistats2] Bug in Top Viewed Articles since bookmark routing changes - https://phabricator.wikimedia.org/T198258#4317348 (10mforns) [20:54:25] ottomata: Can't see you anymore :( [20:54:31] 10Analytics, 10Beta-Cluster-Infrastructure: Disk usage on deployment-kafa-jumbo-* causing alerts - https://phabricator.wikimedia.org/T198262#4317436 (10Krenair) [20:54:51] browser crasehd... [20:57:21] 10Analytics, 10Beta-Cluster-Infrastructure: Disk usage on deployment-kafa-jumbo-* causing alerts - https://phabricator.wikimedia.org/T198262#4317452 (10Krenair) /var/log/kafka is 1.2G on -2 ```root@deployment-kafka-jumbo-2:~# du -hsx /var/log/kafka/* | grep M 257M /var/log/kafka/controller.log.1 257M /var/log... [21:52:59] joal: thanks a LOT for your help. we're running the jobs and it's amazing to be able to do this kind of thing so easily now. big thanks to all of you who worked on mediawiki history data. [21:56:47] leila: \o/ [21:57:23] leila: I just realized that I forgot a WHERE clause in our query on mediawiki-history for revisions: AND NOT revision_is_deleted :) [22:42:10] 10Analytics, 10Analytics-EventLogging: Sample code dialog has extraneous margin - https://phabricator.wikimedia.org/T198269#4317672 (10Krinkle)