[01:17:41] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Fix mediawiki_history_reduced checker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507254 (https://phabricator.wikimedia.org/T222141) (owner: 10Joal)
[01:27:30] <wikibugs>	 (03CR) 10Milimetric: Fix mediawiki_page_history undefined userId (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507258 (https://phabricator.wikimedia.org/T222141) (owner: 10Joal)
[01:37:19] <wikibugs>	 (03CR) 10Milimetric: "Please ping me tomorrow morning on IRC so we can talk about this, as early as you want.  I'll be up really early." (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507260 (https://phabricator.wikimedia.org/T222141) (owner: 10Joal)
[01:41:48] <wikibugs>	 (03CR) 10Milimetric: Fix mediawiki-history user event join (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504834 (owner: 10Joal)
[04:27:24] <groceryheist>	 so next I want to lookup geocodes for pages.  Does wikidata seem like a reasonable way to do this or should I regex the page texts?
[04:54:09] <wikibugs>	 10Analytics, 10Analytics-Cluster: Upgrade Spark to 2.4.2 - https://phabricator.wikimedia.org/T222253 (10Groceryheist)
[05:14:19] <groceryheist>	 I'm getting a weird error in Spark in Swap
[05:14:37] <groceryheist>	 I'm using pyspark
[05:14:45] <groceryheist>	 importerror: no module named 'pyarrow'
[05:14:55] <groceryheist>	 I think it may be related to using a udf
[05:15:05] <groceryheist>	 and the problem might be that pyarrow isn't installed on worker nodes.
[05:15:15] <groceryheist>	 It is available in the notebook
[05:15:28] <groceryheist>	 https://stackoverflow.com/questions/51084514/apply-function-per-group-in-pyspark-pandas-udf-no-module-named-pyarrow
[06:02:12] <elukey>	 groceryheist: hi! Today is bank holiday for most of the europeans, would you mind to open a task tagging "Analytics" ?
[06:02:31] <elukey>	 I'll make sure to work on it asap (or Andrew)
[06:05:41] * elukey afk!
[06:42:24] <groceryheist>	 ok elukey
[06:48:33] <wikibugs>	 10Analytics, 10Analytics-Cluster: Pyspark on SWAP: Py4JJavaError: Import Error: no module named pyarrow - https://phabricator.wikimedia.org/T222254 (10Groceryheist)
[10:51:10] <wikibugs>	 10Analytics, 10Analytics-Cluster: Pyspark on SWAP: Py4JJavaError: Import Error: no module named pyarrow - https://phabricator.wikimedia.org/T222254 (10elukey) https://phabricator.wikimedia.org/T202812 is related to a previous request, maybe it helps! Going to check tomorrow :)
[12:08:56] <wikibugs>	 (03PS1) 10Joal: Fix HdfsUtil forgotten rename [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507549
[12:10:08] <joal>	 Heya team - When somebody gets online, Would you mind reviewing --^ ?
[12:10:23] <joal>	 sqoop is gently failing because of that :(
[12:35:05] <wikibugs>	 10Analytics, 10Performance-Team: Plan navtiming data release - https://phabricator.wikimedia.org/T214925 (10Gilles) 05Open→03Stalled p:05Normal→03Low
[12:51:17] <milimetric>	 joal: howdy
[12:52:33] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] "Sorry I missed this the first time!  I did spot check manually but it's hard to find stuff like this." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507549 (owner: 10Joal)
[12:53:22] <milimetric>	 I'm here to help with the history stuff, priority 1 for me
[12:54:05] <milimetric>	 oh no!  it's May Day, forgot, sorry, enjoy!
[13:05:52] <wikibugs>	 10Analytics, 10Research, 10Article-Recommendation, 10Patch-For-Review: Generate article recommendations in Hadoop for use in production - https://phabricator.wikimedia.org/T210844 (10bmansurov) Thanks, @Ottomata!
[13:28:10] <milimetric>	 joal: I can't tell if you're working or not, I was going to deploy https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/507549/ and restart sqooping
[13:28:18] <milimetric>	 but don't want to conflict with you
[13:28:24] <joal>	 Hi milimetric 
[13:28:40] <joal>	 please excuse me, I missed the previous ping :)
[13:29:02] <joal>	 milimetric: I'm currently preparing the patch for mediawiki-history - Found the issues with reduced
[13:29:29] <joal>	 If you don't mind, I'll finish this (I'll need some input from you), and we can review, merge and deploy after?
[13:29:39] <milimetric>	 joal: I'm here in any way you need me
[13:29:54] <milimetric>	 let me know if you need anything like food delivery :)
[13:29:55] <joal>	 Thanks milimetric :)
[13:30:32] <joal>	 milimetric: let's batcave for a minute so that I tell you my wonders - It'll give you some time to think about them before I need an answer :)
[13:31:02] <joal>	 ottomata: Actually, could you manually kill the currently running sqoop job?
[13:31:04] <milimetric>	 omw
[13:31:24] <ottomata>	 eh?
[13:31:43] <ottomata>	 sqoop-mediawiki-monthly-2019-04-frwiki.pagelinks ?
[13:31:52] <ottomata>	 joal:  ?
[13:31:59] <joal>	 ottomata: the main python job on an-coord1001
[13:32:30] <ottomata>	 oook, just kill it?
[13:32:38] <joal>	 yesp lease - it fails
[13:34:06] <ottomata>	 joal the one that was writing to your user diir?
[13:34:14] <ottomata>	 i hope so cause i killed it :)
[13:34:36] <joal>	 ottomata: MEEEH?
[13:38:41] <ottomata>	 ?
[13:39:53] <joal>	 I was not supposed to write in my folder (
[13:40:19] <joal>	 ottomata: it was supposed to be the real prod one
[13:40:39] <milimetric>	 https://github.com/wikimedia/puppet/blob/c6a9b608f33f77e38aaedee3a4c36989b7fdf78f/modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-private.sh.erb
[13:40:43] <milimetric>	 https://github.com/wikimedia/puppet/blob/c6a9b608f33f77e38aaedee3a4c36989b7fdf78f/modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki-production.sh.erb
[13:40:48] <milimetric>	 https://github.com/wikimedia/puppet/blob/c6a9b608f33f77e38aaedee3a4c36989b7fdf78f/modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki.sh.erb
[13:40:52] <joal>	 thanks milimetric 
[13:40:53] <milimetric>	 those three ottomata 
[13:41:43] <ottomata>	 hmm maybe i killed a few
[13:41:52] <ottomata>	 but there were a bunch that were running since march 08?
[13:42:02] <ottomata>	 e.g.
[13:42:09] <ottomata>	  /usr/bin/python3 /home/joal/code/refinery/bin/sqoop-mediawiki-tables --job-name sqoop-mediawiki-monthly-2019-02 --labsdb --output-dir /user/joal/wmf/data/raw/mediawiki/tables --wiki-file /mnt/hdfs/wmf/refinery/current/static_data/mediawiki/grouped_wikis/labs_grouped_wikis.csv --tables archive,change_tag,change_tag_def,ipblocks,ipblocks_restrictions,logging,page,pagelinks,redirect,revision,user,user_groups --user
[13:42:09] <ottomata>	 s53272 --password-file /user/hdfs/mysql-analytics-labsdb-client-pw.txt --from-timestamp 20010101000000 --to-timestamp 20190301000000 --partition-name snapshot --partition-value 2019-02 --mappers 64 --processors 16 --output-format parquet --log-file /var/log/refinery/sqoop-mediawiki.test-labsdb1012.20190308.log
[13:42:37] <ottomata>	 there are no current sqoop jobs running 
[13:42:50] <ottomata>	 processes*
[13:42:52] <ottomata>	 on an-coord1001
[13:45:12] <joal>	 ottomata: no sqoop maybe, but the python running the thing I think so
[13:45:44] <ottomata>	 ?
[13:46:41] <joal>	 ottomata: https://github.com/wikimedia/puppet/blob/c6a9b608f33f77e38aaedee3a4c36989b7fdf78f/modules/profile/templates/analytics/refinery/job/refinery-sqoop-mediawiki.sh.erb should be running
[13:47:14] <ottomata>	 i killed it all
[13:47:15] <ottomata>	 ?
[13:47:22] <ottomata>	 there's no sqoop running on an-coord1001
[13:47:23] <joal>	 yes please
[13:47:34] <ottomata>	 i just killed all the sqoop there I saw
[13:47:40] <ottomata>	 including a bunch that were writing to your user dir
[13:47:46] <ottomata>	 looked like very old processes?
[13:48:01] <joal>	 probably ottomata - seems weird those very old
[14:13:23] <joal>	 chelsyx: Hi :)
[14:13:53] <milimetric>	 joal: I can go to Scrum of Scrums if you like
[14:14:21] <joal>	 milimetric: That would help a lot - I usually don't go because wednesday is kids' day 
[14:14:34] <milimetric>	 np, I'll go
[14:40:32] <wikibugs>	 (03PS1) 10Joal: Update mediawiki_history after quality phase 1 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507587 (https://phabricator.wikimedia.org/T221824)
[14:41:46] <joal>	 milimetric: --^ please :)
[14:41:56] * milimetric looks
[14:44:29] <wikibugs>	 (03CR) 10Joal: Fix mediawiki-history user event join (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504834 (owner: 10Joal)
[14:44:57] <milimetric>	 joal: what's the plan with the old table?  rename it and point it to an archive of the old snapshots, start fresh with the new schema?
[14:45:20] <joal>	 correct milimetric - Keep old, we'll drop later
[14:45:34] <joal>	 milimetric: I can add the plan to the commit message if you want
[14:45:37] <milimetric>	 but it won't be accessible with the new schema
[14:45:52] <joal>	 correct milimetric - I'll move the tables
[14:46:40] <milimetric>	 that's what I was thinking - move the old snapshots to mediawiki_archive or something like that, rename the table and point it to that archive folder
[14:46:51] <joal>	 yessir
[14:47:16] <joal>	 milimetric: actually might be easier to drop table, move files, recreate and repair table
[14:47:19] <joal>	 but that's details
[14:47:29] <milimetric>	 gotcha
[14:51:12] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata)
[14:52:10] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request - https://phabricator.wikimedia.org/T222268 (10Ottomata)
[14:52:32] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Discovery, 10Discovery-Analysis, and 3 others: Port usage of mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request - https://phabricator.wikimedia.org/T222268 (10Ottomata)
[14:53:05] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata)
[14:53:08] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Discovery, 10Discovery-Analysis, and 3 others: Port usage of mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request - https://phabricator.wikimedia.org/T222268 (10Ottomata)
[14:53:11] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Discovery, and 4 others: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate - https://phabricator.wikimedia.org/T214080 (10Ottomata)
[14:54:56] <wikibugs>	 (03CR) 10Milimetric: Update mediawiki_history after quality phase 1 (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507587 (https://phabricator.wikimedia.org/T221824) (owner: 10Joal)
[14:55:14] <milimetric>	 joal: only one actual question/comment ^
[14:55:45] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Discovery, 10Discovery-Analysis, and 3 others: Port usage of mediawiki_CirrusSearchRequestSet to mediawiki_cirrussearch_request - https://phabricator.wikimedia.org/T222268 (10EBernhardson) The primary user of this data is the oozie job in `wikimedia/discovery/analytics` r...
[14:56:06] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Fix mediawiki-history user event join (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504834 (owner: 10Joal)
[14:56:34] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] "(will merge for now but Joseph and Nuria may still want to sync up on this)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504834 (owner: 10Joal)
[14:56:44] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] Fix mediawiki-history user event join [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/504834 (owner: 10Joal)
[14:57:45] <wikibugs>	 (03CR) 10Milimetric: Update mediawiki_history after quality phase 1 (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507587 (https://phabricator.wikimedia.org/T221824) (owner: 10Joal)
[15:06:55] <wikibugs>	 (03Merged) 10jenkins-bot: Fix mediawiki_history_reduced checker [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507254 (https://phabricator.wikimedia.org/T222141) (owner: 10Joal)
[15:07:18] <wikibugs>	 (03PS2) 10Joal: Update mediawiki_history after quality phase 1 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507587 (https://phabricator.wikimedia.org/T221824)
[15:18:17] <wikibugs>	 (03Abandoned) 10Joal: Fix mediawiki_history eventUserIsAnonymous [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507260 (https://phabricator.wikimedia.org/T222141) (owner: 10Joal)
[15:20:54] <wikibugs>	 (03PS3) 10Joal: Fix mediawiki_page_history userId and anonymous [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507258 (https://phabricator.wikimedia.org/T222141)
[15:22:27] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata) I just tried a version of one of @bd808's [[ https://github.com/bd808/action-api-analytics/blob/master/load-action_ua_hourly.sql | queries ]] to com...
[15:22:44] <wikibugs>	 (03CR) 10Joal: Fix mediawiki_page_history userId and anonymous (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507258 (https://phabricator.wikimedia.org/T222141) (owner: 10Joal)
[15:22:56] <joal>	 milimetric: I hope we're good to go with that :)
[15:26:48] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata) ` select count(*)  FROM wmf_raw.ApiAction WHERE year = ${year}   AND month = ${month}   AND day = ${day}   AND hour = ${hour}  9809748   select coun...
[15:27:09] <milimetric>	 joal: event_user_text is null if the user is not anonymous?
[15:27:15] <milimetric>	 https://gerrit.wikimedia.org/r/#/c/analytics/refinery/source/+/507258/3/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/mediawikihistory/denormalized/MediawikiEvent.scala
[15:28:44] <joal>	 ahah :) no milimetric - userTextHistorical is always defined, and we set current userText to userTextHistorical if user is anonymous (so that the IP is both in historical and current)
[15:29:18] <joal>	 milimetric: https://phabricator.wikimedia.org/T206883
[15:29:23] <milimetric>	 right, I know that
[15:29:41] <milimetric>	 but how does event_user_text get set if it's null here... oh! this is just the event
[15:29:47] <milimetric>	 it gets set by the state?
[15:29:58] <milimetric>	 right
[15:30:53] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Fix mediawiki_page_history userId and anonymous [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507258 (https://phabricator.wikimedia.org/T222141) (owner: 10Joal)
[15:32:11] <milimetric>	 joal: for this https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/507587/1..2/oozie/mediawiki/history/reduced/generate_mediawiki_history_reduced.hql we should talk when you're back
[15:33:54] <joal>	 milimetric: here!
[15:34:13] <joal>	 milimetric: cave?
[15:34:38] <milimetric>	 joal: at sos, but basically shouldn't historical be the default and coalesce to current?  because the event is considered as of that moment in history, no?
[15:34:49] <milimetric>	 it wasn't like that before, so it would be yet another change, sorry to pile it on
[15:34:56] <milimetric>	 and we can keep it like it is for now, change it later
[15:35:27] <joal>	 milimetric: to me, this decision is also based on the wikistats-1 mimic
[15:35:52] <joal>	 milimetric: we prefer to use use current value to provide results similar to WKS1
[15:36:08] <milimetric>	 hm, in this case that was just a limitation of the system, and this is new functionality (lookups in user_groups and such)
[15:36:45] <joal>	 Yessir - IMO we should move toward using historical rather than current, but this would be a big change
[15:37:15] <joal>	 also milimetric, can't be done before page refactor (too many page unlinks as of now I think)
[15:38:11] <milimetric>	 ok joal, will merge as is, and we can revisit later, this one's definitely big enough for a task
[15:38:20] <joal>	 \o/ !!!
[15:38:56] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] "definitely need to update reduced to consider historical values as primary, but we can isolate that change in a later round and consider i" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/507587 (https://phabricator.wikimedia.org/T221824) (owner: 10Joal)
[15:40:54] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Mediawiki-History fixes before deploy - https://phabricator.wikimedia.org/T222141 (10JAllemandou)
[15:41:13] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Mediawiki-History fixes before deploy - https://phabricator.wikimedia.org/T222141 (10JAllemandou) a:03JAllemandou
[15:41:42] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10JAllemandou)
[15:42:59] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: Mediawiki History Release - 2019-04 snapshot - https://phabricator.wikimedia.org/T221824 (10JAllemandou)
[15:47:04] <wikibugs>	 (03PS1) 10Joal: Update changelog.md to version 0.0.88 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507603
[15:48:11] <milimetric>	 joal: I'm gonna jump in the cave early if you need any help with release or anything before standup
[15:48:24] <joal>	 milimetric: OMW :)
[15:48:48] <joal>	 milimetric: if you can double check the changelog patch above--^
[15:48:48] <joal>	 :)
[15:49:30] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] Update changelog.md to version 0.0.88 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/507603 (owner: 10Joal)
[15:50:19] <joal>	 ottomata: there are 2 tasks in ready-to-deploy  - is there anything for me to do with them?
[15:54:30] <wikibugs>	 10Analytics: Use historical fields in history reduced dataset - https://phabricator.wikimedia.org/T222278 (10Milimetric)
[15:55:26] <wikibugs>	 10Analytics, 10Analytics-Cluster: Pyspark on SWAP: Py4JJavaError: Import Error: no module named pyarrow - https://phabricator.wikimedia.org/T222254 (10Groceryheist) @elukey, thanks. It seems like I'm experiencing a regression then. I can work around it for now. See you tomorrow!
[15:55:58] <ottomata>	 joal nope
[15:56:06] <joal>	 k - thanks ottomata :)
[16:03:31] <milimetric>	 nuria: you around today?
[16:08:05] <joal>	 !log refinery-source v0.0.88 released on archiva
[16:08:06] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:35:55] <ottomata>	 bd808:  do the api logs also go to udp2log?
[16:37:16] <ottomata>	 answer: yes.
[16:37:19] <ottomata>	 found it.
[16:43:19] <chelsyx>	 Hi joal :)
[16:43:46] <joal>	 chelsyx: I have questions about your job mimicing the uniquefor-mobiles
[16:44:27] <chelsyx>	 joal: yes
[16:45:10] <joal>	 chelsyx: two things - I wonder about its usage (by-country job should make the job?), and I'd also ask for a change in date - 1st of month is a very busy day for the clustrer, moving the job to the 2nd or 3rd would help a lot :)
[16:46:58] <chelsyx>	 joal: I think you're talking about 'mobile_apps-uniques-by_country-monthly' right?
[16:47:14] <joal>	 chelsyx: yes
[16:48:43] <joal>	 !log Kill oozie mediawiki-history-druid-coord for true (replaced by edit_hourly job)
[16:48:45] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:49:00] <chelsyx>	 joal: we don't use it very frequently now -- we're working on switching to another method similar to the 'last-access' cookie, but use eventlogging , for a more accurate unique device counts for the apps. After that's done, I will stop this job
[16:49:22] <chelsyx>	 joal: And of course, I can move it to later days in the month :)
[16:49:39] <joal>	 Thanks for that chelsyx :)
[16:50:01] <chelsyx>	 joal: As we're speaking, can I ask you a question?
[16:50:07] <joal>	 chelsyx: I'd be interested if there is a measured diff between the per-country and the global result
[16:50:43] <joal>	 chelsyx: For uniques using last-access, we assumed global = Sum(per-country) - This is why I ask :)
[16:50:47] <joal>	 chelsyx: please ask :)
[16:51:02] <chelsyx>	 joal: I haven
[16:51:27] <chelsyx>	 joal: thatI haven't check whether global = Sum(per-country)
[16:51:40] <chelsyx>	 joal: that's a good question :)
[16:52:39] <chelsyx>	 joal: I'm running a query to get pageviews by page_id on enwiki in March, and I also want to get the latest rev_id for each page. But this query seems take forever to run:
[16:52:53] <chelsyx>	 https://www.irccloud.com/pastebin/EXANzA7L/
[16:53:17] <chelsyx>	 do you have any suggestion about how to optimize it?
[16:55:16] <joal>	 chelsyx: looking at all pageviews is huge - putting a threshold and cutting the longtail would help a lot
[16:56:11] <chelsyx>	 joal: I see, I'm thinking about that too
[16:56:39] <joal>	 the other thing I'm not sure of is how the sywstem (hive or spark?) deals with the mediawiki_page partition filter in the join query
[16:57:11] <joal>	 wmf_raw.mediawiki_page is partitioned by wiki_db, so it's important to take advantage of that
[16:57:54] <chelsyx>	 joal: ah, got it. I misses that
[17:00:30] <chelsyx>	 joal: Thanks!
[17:00:37] <joal>	 np chelsyx :)
[17:00:54] <joal>	 Thank you for moving the big job a bit later :)
[17:02:40] <joal>	 !log Deploying refinery using scap
[17:02:42] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:03:56] <chelsyx>	 joal: No problem!
[17:08:20] <joal>	 !log Killing oozie jobs for new deploy: mediawiki-history-denormalize-coord, mediawiki-history-check_denormalize-coord, mediawiki-history-reduced-coord
[17:08:22] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:12:10] <joal>	 !log Moving exisiting mediawiki-history to /wmf/data/wmf/mediawiki/archive folder
[17:12:11] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:18:11] <joal>	 ottomata: Hellooooo :(
[17:18:19] <ottomata>	 hiyyaaa
[17:18:23] <joal>	 ottomata: I'm having an issue with scap deploy :(
[17:18:31] <joal>	 ottomata: 2 hosts failed
[17:19:05] <milimetric>	 chelsyx: I'm curious about your query for my own reasons, I'm playing with it to see if I can make it faster, will post my attempt in a second
[17:19:41] <joal>	 ottomata: https://gist.github.com/jobar/47edfadd421beac4c53ed38c0064efb2
[17:20:05] <joal>	 ottomata: looks like some git-fat path have not been changed to analytics_deploy user I guess :(
[17:20:17] <ottomata>	 hmmm ohhh
[17:20:33] <milimetric>	 yeah, luca just did the analytics_deploy stuff this morning
[17:20:42] <milimetric>	 before he realized he was on vacation :)
[17:21:27] <joal>	 ottomata: I'm assuming I should rollback - correct?
[17:22:31] <ottomata>	 hang on
[17:22:35] <ottomata>	 did it succeed elsewhere?
[17:23:21] <joal>	 ottomata: seems to have succeeded in db1107, an-coord1001, stat1004 and stat1006
[17:23:37] <ottomata>	 hm
[17:23:51] <ottomata>	 joal:  don't rollback then.
[17:24:08] <ottomata>	 i'd say continue and then try to do it again.  i was just able to run the git fat pull command as analytics-deploy on notebook1003.
[17:24:16] <ottomata>	 so, it might be temporary
[17:24:19] <joal>	 hm
[17:24:22] <ottomata>	 or you could rollback and try again
[17:24:32] <ottomata>	 if you rollback, I can try if you like
[17:24:37] <joal>	 ottomata: Which do you prefer?
[17:24:52] <milimetric>	 chelsyx: ok, it seems a little bit faster, but no big gain, maybe it's easier to write separated out like this?
[17:24:55] <milimetric>	 https://www.irccloud.com/pastebin/TETQNogW/
[17:25:50] <joal>	 milimetric: a lot more readable, and enforces partition usage on mediawiki_page --^
[17:25:53] <joal>	 Nice
[17:26:10] <chelsyx>	 milimetric: Thanks for the suggestion! I will try
[17:26:14] <milimetric>	 her original was using the partition in the where clause, that's what I was curious about, if it makes a difference doing it directly
[17:26:16] <milimetric>	 doesn't seem to
[17:26:37] <joal>	 not sure :S
[17:26:44] <milimetric>	 Hive seems to do the same amount of work, yeah, I wasn't sure either
[17:26:47] <joal>	 ottomata: ping ?
[17:27:08] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata) Hm ok, I don't fully know what is going on here, but I found some api requests that are missing from ApiAction.  I focused on one specifically, and...
[17:27:16] <ottomata>	 sorry was finishign writing ^
[17:27:25] <ottomata>	 joal go ahead  and rollback, i'll tyr
[17:27:27] <ottomata>	 try
[17:27:28] <joal>	 sorry ottomata :) Was wondering :)
[17:27:35] <joal>	 ok, rollbacking
[17:28:02] <joal>	 ottomata:  No space left on device
[17:28:16] <joal>	 notebook1003 and 4
[17:28:22] <ottomata>	 oh ho ho
[17:28:25] <ottomata>	 that is probslby the problem then'
[17:28:27] <ottomata>	 grrr
[17:28:30] <joal>	 :S
[17:28:45] <joal>	 I thought using a small number of cached revs should have solved
[17:28:52] <ottomata>	 31G	./dsaez
[17:29:19] <ottomata>	 9.2G	./neilpquinn-wmf
[17:29:23] <ottomata>	 16G	./piccardi
[17:29:43] <joal>	 ottomata: same partition?
[17:29:50] <ottomata>	 ya 
[17:29:58] <joal>	 mwarf
[17:30:04] <ottomata>	 notebook1004
[17:30:05] <ottomata>	 11G	./ebernhardson
[17:30:15] <ottomata>	 41G	./nathante
[17:32:13] <ebernhardson>	 :(
[17:32:17] <ebernhardson>	 lemme see what i'm doing there...
[17:32:42] <ebernhardson>	 virtualenv's :P
[17:33:03] <joal>	 Wow - 11G of virtualenvs !! /me has a love-hate relation with a python
[17:33:31] * joal has a hate-hate relationship with maven though
[17:33:45] <ebernhardson>	 spacy is 376MB, tensorflow is 371MB, scipy is 126MB, numpy is 71MB.  I have these installed in multiple venv's
[17:34:18] <ebernhardson>	 i can probably delete a few though, some are simply backups before installing other stuff incase it al lbreaks
[17:35:13] <ebernhardson>	 my local maven repo is only 3.6G :)
[17:35:33] <ebernhardson>	 funny thing that probably is all java bytecode, whereas i'm sure there are datasets for nlp and whatnot in my python virtualenv's
[17:36:13] <joal>	 agreed ebernhardson 
[17:37:50] <joal>	 !log Recreate wmf.mediawiki_history (+page and user) and  wmf.mediawiki_history_archive (with old data)
[17:37:52] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:39:29] <ottomata>	 ok joal  i freed some space
[17:39:34] <ottomata>	 i'm assuming that was your problem
[17:39:36] <ottomata>	 so try the deploy again
[17:39:40] <joal>	 Many thanks ottomata - Trying again
[17:40:23] <joal>	 !log deploy refinery using scap after failed attemp
[17:40:25] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:41:04] <ottomata>	 actualy worried about notebook1003, there is only 11G there atm
[17:41:08] <ottomata>	 free
[17:42:03] <ottomata>	 joal do you have anything there ytou can free up
[17:42:06] <ottomata>	 you are using 2.9G
[17:42:12] <joal>	 Checking ottomata 
[17:45:38] <joal>	 ottomata: deploy successfull - Plenty thanks!
[17:48:29] <ottomata>	 k great
[17:48:34] <joal>	 ottomata: almost everything in my home comes from the venv folder
[17:48:40] <ottomata>	 k
[17:48:47] <joal>	 ottomata: is it theo ne used by jupyther?
[17:48:51] <ottomata>	 yes
[17:48:51] <ebernhardson>	 i shed mine from 11G to 2.5G, hopefully ok?
[17:48:55] <ottomata>	 thanks ebernhardson !
[17:48:59] <ottomata>	 much appreciataed
[17:49:02] <ebernhardson>	 np
[17:49:02] <joal>	 thanks ebernhardson :)
[17:49:14] <joal>	 ottomata: ok - I guess I can't do a lot :S
[17:49:35] <ottomata>	 ya no worried
[17:49:47] <ottomata>	 yours is fine, was just crossing fingers for deploy
[17:50:04] <ebernhardson>	 if we had modern shared versions of spacy, tensorflow, scipy and numpy maybe each user wouldn't need to install them :) unfortunately the deb packages are ancient
[17:50:45] <joal>	 ottomata: quick question - do we on purpose not deploy jars on notebook1003?
[17:50:53] <ottomata>	 no?
[17:51:14] <joal>	 just checked out of curiosity, and the size of refinery-job-0.0.88.jar is 74 bytes
[17:51:25] <ottomata>	 then git fat pull failed
[17:51:31] <joal>	 right
[17:51:37] <joal>	 but scap didn't
[17:51:46] <joal>	 maybe because of re-deploy, but seems weird
[17:52:07] <joal>	 will double check on an-coord1001 (refine), and on stat1004 before deploying to hhdfs
[17:52:11] <joal>	 ottomata: --^
[17:52:13] <ottomata>	 k
[17:52:28] <ottomata>	 hm, runnign git fat pull manually on notebook1003 is pulling them down...
[17:52:40] <ottomata>	 joal we should probably do a jar version prune again soon
[17:52:52] <joal>	 yessir
[17:52:56] <joal>	 very much agreed
[17:53:00] <joal>	 ok on an-coord1001
[17:53:08] <ottomata>	 k
[17:53:14] <ottomata>	 yeah maybe it got stuck because of deploy fail
[17:53:55] <joal>	 I assume so, not the first time it happens - when deploy fails because of git-fat (space), redeploy succeeds immediatly since code is already there
[17:54:00] <joal>	 But mo git-fat
[17:55:05] <joal>	 !log deploying refinery onto HDFS
[17:55:07] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[17:57:10] <joal>	 ottomata: asking permission to manually start sqoop
[17:57:17] <ottomata>	 sure!
[17:57:22] <joal>	 thanks :)
[17:59:02] <joal>	 !log Starting a manual run of sqoop
[17:59:03] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:01:51] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Nuria) >I think mediawiki_api_request is correct. Is it possible that LOTS of data is somehow missing from ApiAction, and has been? We can verify this by coun...
[18:03:15] <nuria>	 ottomata: re: api requests
[18:03:48] <ottomata>	 nuria:  good idea w webrequest
[18:03:50] <ottomata>	 will look
[18:08:13] <joal>	 !log Manually killed sqoop-production (comment and actor) to have it done after the current manual labs run
[18:08:14] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:21:27] <joal>	 ottomata: have you silenced a systemd timer alarm?
[18:21:49] <wikibugs>	 10Analytics-Cluster, 10Analytics-Kanban: refinery-sqoop-mediawiki-production on an-coord1001 needed restart - https://phabricator.wikimedia.org/T222294 (10crusnov)
[18:22:20] <joal>	 !log Confirming that sqoop-private (cu_changes) will run automatically tonight (2nd of the month at 00:00) - nothing needed
[18:22:21] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:23:51] <nuria>	 ottomata: ok, i am going to try the backfilling in a bit now that i have couple hours free, let me know if you want help looking ta apiaction
[18:23:55] <nuria>	 *at apiaction
[18:25:39] <joal>	 !log restarted oozie jobs mediawiki-history-denormalize-coord, mediawiki-history-check_denormalize-coord and mediawiki-history-reduced-coord
[18:25:41] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:32:50] <joal>	 Ok team, all in place so far - sqoop-labsdb running, will launch manual run of sqoop-prod (actor/comment) tomorrow morning after having double checked labsdb finiched ok
[18:34:04] <joal>	 Oozie jobs restarted, old data moved to _archive folders/tables, comparison snapshot for checker currently being computed (I'll put it in production folder once done and vetted manually)
[18:34:21] <ottomata>	 :)
[18:34:23] <joal>	 All of that is noted here: https://etherpad.wikimedia.org/p/analytics-weekly-train
[18:35:12] <joal>	 Gone for diner, will be back checking jobs after
[18:41:18] <ottomata>	 nuria:  thanks, am poking...
[18:41:35] <ottomata>	 the request i am investigating with is in webrequest
[18:41:50] <ottomata>	 would be nice to know if it was in  Kafka, but reading avro out of kafka is not fun
[18:47:22] <nuria>	 ottomata: mmm,  does camu import this data at all?
[18:47:50] <ottomata>	 yes
[18:48:04] <ottomata>	 the data that caamus imports is the ApiAction aable
[18:48:15] <ottomata>	 it'd be really nice to have a reliable x-request-id!!!
[18:48:23] <nuria>	 ottomata:  I see, kafka-> api action table directly
[18:48:30] <nuria>	 ottomata: a session?
[18:48:32] <ottomata>	 via camus yes, since it s avro
[18:48:34] <nuria>	 ottomata: jajajaja
[18:48:40] <ottomata>	 i'm looking for  agood join field
[18:49:19] <nuria>	 ottomata: md5(ip+raw ua ) i think would be your best bet
[18:49:26] <ottomata>	 hmmm
[18:49:34] <ottomata>	 not  a bad ideaa
[18:49:34] <ottomata>	 ok
[18:49:50] <ottomata>	 i'll just join on both of those
[18:50:11] <nuria>	 ottomata: and if does not work add date up to the hour
[18:55:31] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Tgr) AIUI the webrequest table only contains GET requests but includes requests that Varnish serves from cache (Varnish caching is not typical for the Action...
[19:13:24] <nuria>	 ottomata: i am getting pertinent files for backfilling  from hadoop, but after there is no way to [process them other than in teh eL box right?
[19:13:30] <nuria>	 *the eventlogging box
[19:14:31] <ottomata>	 you can probably do it wherever, as long sa eventlogging can run.
[19:15:01] <nuria>	 ottomata: but that is the thing, it can only run  on EL machine cause that is where its deps are installed right?
[19:15:11] <ottomata>	 hmmmm
[19:15:14] <ottomata>	 yes.
[19:16:03] <elukey>	 ottomata: o/ - whenever you have time, I think I found a way to fix the log4j logging https://gerrit.wikimedia.org/r/#/c/operations/puppet/cdh/+/507630/
[19:16:14] <elukey>	 let me know if it makes sense
[19:16:35] <ottomata>	 +1!
[19:16:40] <ottomata>	 nuria yeah hm.  
[19:16:55] <ottomata>	 we could probably include eventlogging class on stat1007 or something
[19:17:10] <elukey>	 going to roll it out now so we stop getting broken outputs
[19:18:26] <ottomata>	 cool elukey  thnaks!
[19:18:36] <ottomata>	 nuria: there is an eventlogging::dependencies class.  
[19:18:39] <elukey>	 sorry for the spam in the commands :(
[19:18:52] <nuria>	 elukey: yay 
[19:19:04] <nuria>	 elukey: kind of late eh? this CAN WAIT
[19:19:49] <elukey>	 nuria: I knowww :P
[19:20:29] <nuria>	 ottomata: installing a running version of EL in the stats machines to backfill seems kind of involved....maybe i  am chicken
[19:21:00] <ottomata>	 i think its pretty easy actulaly!  i'm not going to include eventlogging codebase itself
[19:21:07] <ottomata>	 you can clone that in your home dir
[19:21:08] <ottomata>	 but
[19:21:21] <ottomata>	 the deps are easy, since they are all debs
[19:21:23] <ottomata>	 and in their own class
[19:22:19] <elukey>	 fixed! just tested on stat1007.. puppet is rolling it out everywhere
[19:22:28] <elukey>	 ok now I feel better :D
[19:22:34] <ottomata>	 greta!
[19:22:36] <elukey>	 o/
[19:22:46] <elukey>	 (going afk otherwise Marika will probably kill me :P)
[19:31:17] <ottomata>	 nuria:  if you like
[19:31:20] <ottomata>	 on stat1004
[19:31:26] <ottomata>	 export PYTHONPATH=/home/otto/eventlogging
[19:31:37] <ottomata>	 export PATH="$PATH:/home/otto/eventlogging/bin"
[19:31:54] <ottomata>	 then
[19:31:55] <ottomata>	 eventlogging-processor --help
[19:31:56] <ottomata>	 :)
[19:35:39] <nuria>	 ottomata: WOW
[19:35:56] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata) Haven't yet counted totals (a bit hard and expensive to join properly here), but as expected the request I'm looking at is also in webrequest:  `lan...
[19:36:40] <nuria>	 ottomata: same for 1007 right? (minus your checkout)
[19:37:40] <ottomata>	 you know it didn't install them on stat1007...
[19:38:08] <ottomata>	 i think maybe they were already there
[19:39:23] <ottomata>	 grr stat1005 is buster and is busting this up
[19:39:31] <ottomata>	 since deps aren't avaail for buster.
[19:40:23] <nuria>	 ottomata:  we can fake upload an empty deb , i know, you love this iddea
[19:40:35] <ottomata>	 na we don't need on stat1005
[19:40:37] <joal>	 milimetric: I'm about to leave for tonight - Does the state of affair look ok for you?
[19:40:40] <ottomata>	 we've aalreadya got a conditional for buster
[19:42:44] <milimetric>	 joal, all jobs make sense, if somehow sqoop finishes early before I go to sleep I’ll launch comment/actor
[19:42:59] <joal>	 milimetric: ack 
[19:43:22] <joal>	 milimetric: it is supposed to finish in ~10 hours - possibly too late for you, but just in time for my morning ;)
[19:43:33] <joal>	 Ok gone for tonight tema - see you tomorrow
[19:46:20] <milimetric>	 :) nite
[19:47:36] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Nuria) I think there is no need to join, plain counts in webrequest and apiAction (old table) should match closely for api requests with MISS status in webreq...
[19:48:14] <nuria>	 ottomata: so, to clarify, we DO have these deps on stats1007, correct?
[19:49:06] <wikibugs>	 10Analytics: Upgrade pandas in spark SWAP notebooks - https://phabricator.wikimedia.org/T222301 (10Groceryheist)
[19:49:23] <ottomata>	 i thikn so yess nuria
[19:50:08] <ottomata>	 nuria I have eventlogging cloned there in same place
[19:50:14] <ottomata>	 same exports should work there too
[19:50:36] <nuria>	 ottomata: k, just clone it but will try , let me clean up files a bit cause they have a bunch of whitelines
[19:52:28] <joal>	 ottomata: sqoop-production got restarted after I killed it manually
[19:52:32] <joal>	 ottomata: any idea?
[19:52:41] <ottomata>	 hm, no haven't touched it
[19:52:46] <joal>	 :(
[19:52:48] <joal>	 ok
[19:53:12] <joal>	 ottomata: I'm assuming it'll raise an alarm in ops chan - is there a way for us to silent it?
[19:53:13] <ottomata>	 joal wheree are you looking?
[19:53:16] <nuria>	 joal: the timer would have restarted it if you did not disabled it right?
[19:53:34] <joal>	 ottomata: I noticed jobs on the cluster, then checked on an-coord1001
[19:53:40] <joal>	 I just manually killed it again
[19:53:41] <nuria>	 ottomata: does this look good?
[19:53:45] <nuria>	 https://www.irccloud.com/pastebin/yF9aO4Ju/
[19:53:56] <nuria>	 joal: if it is a systemd timer i think it will start again
[19:54:05] <joal>	 nuria: really ?
[19:54:14] <joal>	 Arf - not good 
[19:54:26] <nuria>	 joal: ya,  you need to disable the timer no? 
[19:54:55] <joal>	 Yeah you're right
[19:54:57] <joal>	 truing so
[19:55:31] <nuria>	 joal: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers#Disable_a_Systemd_timer
[19:55:39] <joal>	 nuria: I was reading that
[19:56:49] <joal>	 Looks like I can't - need root powa
[19:56:54] <joal>	 ottomata: can ou help please?
[19:57:11] <joal>	 ottomata: from an-coord1001: sudo systemctl disable refinery-sqoop-mediawiki-production
[19:58:15] <icinga-wm>	 PROBLEM - Check the last execution of refinery-sqoop-mediawiki-production on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit refinery-sqoop-mediawiki-production
[19:58:40] <ottomata>	 !log sudo systemctl disable refinery-sqoop-mediawiki-production
[19:58:41] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[19:58:42] <ottomata>	 joal: done
[19:58:49] <joal>	 Thanks ottomata 
[19:59:17] <nuria>	 joal: i think we should all have permits to disable/enable these, let's talk to luca tomorrow
[19:59:23] <joal>	 yes
[19:59:59] <ottomata>	 nuria:  i don't think you wanat to quote all of the args in one
[20:00:19] <ottomata>	 e.g. stdin:// should be separate from the kafkaa ones
[20:00:37] <nuria>	 ottomata: let me look in el box
[20:00:40] <ottomata>	 aalso
[20:00:44] <nuria>	 ottomata: so i understand what you mena
[20:00:46] <ottomata>	 the el one uses just kafka://
[20:00:46] <nuria>	 *mean
[20:00:49] <ottomata>	 not kafka-confluent://
[20:00:53] <ottomata>	 nuria i meana
[20:01:02] <ottomata>	 you have single quotes starting at 'stdin:// 
[20:01:09] <ottomata>	 and ending after the kafka output uri
[20:01:18] <ottomata>	 that will feed it to python as a single arg
[20:01:41] <ottomata>	 you'll only need to put single quotes around the kafka uri
[20:01:46] <ottomata>	 since it uses e.g. & chars
[20:02:28] <joal>	 I'm very sorry ottomata, reading the doc again it looks like I forgot a step - sudo systemctl stop refinery-sqoop-mediawiki-production
[20:02:34] <joal>	 Then I promise I stop bothering
[20:02:57] <ottomata>	 !log sudo systemctl stop refinery-sqoop-mediawiki-production
[20:02:58] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[20:03:15] <icinga-wm>	 ACKNOWLEDGEMENT - Check the last execution of refinery-sqoop-mediawiki-production on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit refinery-sqoop-mediawiki-production ottomata systemd timer stopped manually.
[20:04:08] <nuria>	 ottomata: 
[20:04:12] <nuria>	 https://www.irccloud.com/pastebin/UxQlZe20/
[20:04:54] <ottomata>	 should be good I think nuria
[20:05:04] <ottomata>	 nuria btw, you won't be backfilling into eventlogging-valid-mixed
[20:05:05] <ottomata>	 is that ok?
[20:05:13] <nuria>	 ottomata: ya
[20:05:18] <ottomata>	 k
[20:05:20] <nuria>	 ottomata: i thought about that 
[20:05:40] <nuria>	 ottomata: but as i was saying i think we should start thinking about it being kaput
[20:06:05] <nuria>	 ottomata: none of the salt+sanitization work marcel did is applying there for example
[20:06:17] <nuria>	 ottomata: i mean some of it is but mostly not
[20:07:07] <ottomata>	 :)
[20:10:18] <ottomata>	 hm nuria  i must be selecting from webrequest wrong
[20:10:32] <ottomata>	 maybe its this miss.
[20:10:32] <ottomata>	 hm
[20:11:04] <nuria>	 ottomata: batcave? 
[20:11:21] <ottomata>	 naw hang on
[20:14:12] <ottomata>	 nuria:  i selected some distinct cache_statuses for an hour of %api.php%
[20:14:15] <ottomata>	 hit-remote
[20:14:15] <ottomata>	 hit-local
[20:14:15] <ottomata>	 miss
[20:14:15] <ottomata>	 int-local
[20:14:15] <ottomata>	 -
[20:14:15] <ottomata>	 pass
[20:14:15] <ottomata>	 hit-front
[20:14:16] <ottomata>	 int-front
[20:14:24] <ottomata>	 so maybe coungint where NOT LIKE 'hit%' ?
[20:16:10] <nuria>	 ottomata: right I forgot about this , what is "int-local?"
[20:16:19] <nuria>	 ottomata: and "int-front?"
[20:16:19] <ottomata>	 dunno!
[20:16:26] <nuria>	 ottomata: one sec 
[20:19:37] <nuria>	 ottomata: ok, read all docs but this  int-front does not appear
[20:20:03] <nuria>	 ottomata: kind of seems like internal-front 
[20:20:34] <ottomata>	 but still missing some
[20:20:46] <nuria>	 ottomata: so i would go for pass and expect those match api action (old)
[20:20:46] <ottomata>	 and with no cache_status filter
[20:20:48] <ottomata>	 too many
[20:20:55] <ottomata>	 hmm
[20:20:57] <ottomata>	 checking that
[20:25:33] <ottomata>	 nuria:  with just pass
[20:25:36] <ottomata>	 still more than ApiAction
[20:25:54] <nuria>	 ottomata: super interesting interesting
[20:26:45] <nuria>	 ottomata: now i would just group by method or similar and see (for an hour only using pass data) if  apttern emerges
[20:26:49] <nuria>	 ottomata: also, the reverse
[20:26:58] <nuria>	 ottomata: do all apiaction have pass?
[20:27:18] <ottomata>	 i'm grouping by cache_status now
[20:27:22] <ottomata>	 will do method next
[20:39:27] <nuria>	 ottomata: k let me know
[20:39:48] <ottomata>	 was querying a daya, should have done an hour
[20:40:16] <nuria>	 ottomata: ya, couple hours for two different days are enough
[21:02:37] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Ottomata) Gathered some counts:  ` set hivevar:year=2019; set hivevar:month=4; set hivevar:day=30; set hivevar:hour=0; `  **wmf_raw.ApiAction** `9809748`  **e...
[22:46:55] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10EventBus: Port usage of mediawiki_ApiAction to mediawiki_api_request - https://phabricator.wikimedia.org/T222267 (10Tgr) Not counting all the hit-* requests is expected since those never reach the appservers and ApiAction is logged by the appservers. (Not sure what int-* is...