[04:49:06] Analytics, Pageviews-API: Add support for outreachwiki to pageviews API - https://phabricator.wikimedia.org/T132313#2194166 (MusikAnimal) [07:22:51] (PS2) Amire80: Add a script for checking number of pages published despite failures [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282312 (https://phabricator.wikimedia.org/T127283) [07:23:09] (PS3) Amire80: Add sorted errors [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282228 (https://phabricator.wikimedia.org/T127283) [07:47:41] (CR) KartikMistry: Add sorted errors (1 comment) [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282228 (https://phabricator.wikimedia.org/T127283) (owner: Amire80) [07:54:39] (CR) Nikerabbit: Add sorted errors (2 comments) [analytics/limn-language-data] - https://gerrit.wikimedia.org/r/282228 (https://phabricator.wikimedia.org/T127283) (owner: Amire80) [08:36:43] Hi elukey ! [08:37:43] joal: gooooooood morning :) [08:37:54] Had a good weekend elukey ? [08:38:17] yessss.. good weather and good food :) [08:38:38] Awesome :)hn [08:39:44] joal: and you?? [08:40:35] Good as well, I finally managed to mow the lawn ;) [08:42:25] elukey: I'm currently preparing refinery-source release & deploy [08:42:40] elukey: Will need your help later on for reviews :) [08:44:54] joal: sure! [08:45:07] did you read my message about https://phabricator.wikimedia.org/T132256#2192798 ? [08:45:22] super weird [08:45:34] * joal reads [08:46:11] hm, weird indeed [08:46:24] elukey: Do you think we use them to much ? [08:48:21] joal: nah but there might be some dc-ops work to do like it was suggested.. just wanted to give you an heads up, let's check for other weirdness from now on to spot a trend.. [08:48:41] (hopefully there's none but you know that I am always paranoid :P) [08:48:53] elukey: truth is out there ! [08:50:18] (PS1) Joal: Update changelog for release v0.0.30 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282660 [08:50:22] elukey: reviews start :) [08:55:11] joal: looking! Quick question in the meantime: https://logstash.wikimedia.org/#/dashboard/elasticsearch/restbase -> analytics.wikimedia.org shows 500s [08:55:30] is it something that we already know or a new thing? [08:56:07] elukey: it happens, it's due to timeouts from assandra [08:56:35] elukey: gwicke confirmed me it was mostly related to 'big' datasets asked (multiple month of data) [08:57:19] :( [08:57:37] elukey: I'm really looking forward to SSDs [09:00:23] might be interesting to build some docs around how to debug these issues [09:00:35] joal: anyhoww, CR looks good [09:00:52] (not sure if I need to nod or just observe :) [09:00:59] elukey: unfortunately no real debug way - Or limitting request size [09:01:12] elukey: if you can +2 would be awesome :) [09:01:37] (CR) Elukey: [C: 2] Update changelog for release v0.0.30 [analytics/refinery/source] - https://gerrit.wikimedia.org/r/282660 (owner: Joal) [09:01:46] Thx ! [09:13:02] !log Releasing refinery-source v0.0.30 to archiva [09:13:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [09:48:39] (PS1) Joal: Add v0.0.30 jars and update links [analytics/refinery] - https://gerrit.wikimedia.org/r/282663 [09:48:45] elukey: --^ [09:50:19] (PS1) Joal: Bump jar version in refine and cassandra jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/282664 [09:50:25] elukey: and --^ [09:50:30] elukey: Please :) [09:53:29] (CR) Elukey: [C: 2] Add v0.0.30 jars and update links [analytics/refinery] - https://gerrit.wikimedia.org/r/282663 (owner: Joal) [09:53:38] (CR) Elukey: [V: 2] Add v0.0.30 jars and update links [analytics/refinery] - https://gerrit.wikimedia.org/r/282663 (owner: Joal) [09:53:51] (CR) Elukey: [C: 2 V: 2] Bump jar version in refine and cassandra jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/282664 (owner: Joal) [09:54:06] joal: --^ lgtm [09:54:15] Thanks Mate :) [10:04:37] Analytics, Operations: kafkatee cronspam from oxygen - https://phabricator.wikimedia.org/T132322#2194403 (elukey) [10:09:17] Analytics, Discovery, Maps, RESTBase-Cassandra, Patch-For-Review: Investigate and implement possible simplification of Cassandra Logstash filtering - https://phabricator.wikimedia.org/T130861#2194418 (jstenval) [10:14:46] sigh kafkatee has logrotate/syslog configs in the debian package [10:14:56] so when rsyslog is not installed, cronspam [10:15:51] hm, I think I don't know enough to understand elukey :) [10:17:09] joal: no no nothing really important, basically kafkatee is package (as varnishkafka) with a config like https://github.com/wikimedia/analytics-kafkatee/blob/debian/debian/kafkatee.logrotate [10:17:39] so when you install the debian package you get the files installed automatically [10:17:48] and in the files it is mentioned service rsyslog reload [10:18:13] on oxygen.e.wmnet we use kafkatee, but rsyslog is not installed [10:18:28] (basically it is used by ops to grab logs from kafka) [10:18:38] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Create basic/high-level Kabana (dashboard) documentation - https://phabricator.wikimedia.org/T132323#2194419 (Aklapper) [10:18:42] (in a programatic way) [10:18:47] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Create basic/high-level Kabana (dashboard) documentation - https://phabricator.wikimedia.org/T132323#2194419 (Aklapper) p:Triage>Normal [10:19:18] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Create basic/high-level Kabana (dashboard) documentation - https://phabricator.wikimedia.org/T132323#2194419 (Aklapper) [10:19:20] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Play with Bitergia's Kabana UI (which might potential replace our current UI on korma.wmflabs.org) - https://phabricator.wikimedia.org/T127078#2031579 (Aklapper) [10:19:39] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Play with Bitergia's Kabana UI (which might potential replace our current UI on korma.wmflabs.org) - https://phabricator.wikimedia.org/T127078#2031579 (Aklapper) Open>stalled [10:19:42] elukey: Ah right [10:21:45] !log deploying refinery from tin [10:21:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [10:24:20] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): Play with Bitergia's Kabana UI (which might potential replace our current UI on korma.wmflabs.org) - https://phabricator.wikimedia.org/T127078#2194443 (Aklapper) Random notes I took in our meeting on 2016-04-08: * Some stuff in Kibana... [10:30:50] !log Deploying refinery on HDFS [10:30:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [10:43:49] Analytics, Operations: kafkatee cronspam from oxygen - https://phabricator.wikimedia.org/T132322#2194462 (elukey) [11:14:58] Analytics-Kanban: Update reportupdater's documentation - https://phabricator.wikimedia.org/T132326#2194522 (mforns) [11:18:50] Analytics-Kanban, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, Patch-For-Review, and 2 others: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2194546 (Tbayer) @nuria A friendly reminder about your kind offer from Friday on IRC to paste a webrequest qu... [11:40:17] lunch! [11:52:30] !log Restart refine job after deploy [11:52:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [12:14:26] Analytics-Kanban, Wikipedia-iOS-App-Product-Backlog, iOS-app-feature-Analytics, Patch-For-Review, and 2 others: Invalid pageview data for iOS app - https://phabricator.wikimedia.org/T131824#2194636 (JAllemandou) @Tbayer : You'll find extracted iOs pageview data from March 10th 15:00 UTC to April 11... [12:38:53] Analytics-Tech-community-metrics, Developer-Relations (Apr-Jun-2016): top-contributors should have real names for the main contributors - https://phabricator.wikimedia.org/T124346#2194665 (Aklapper) Open>Resolved First push has gone live. I've pushed a [[ https://github.com/Bitergia/mediawiki-iden... [13:47:26] Analytics, RESTBase, Services: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2194801 (mobrovac) p:Normal>High Raised the priority to High as we should settle on this ASAP. It's been around for a while now without any... [13:47:41] Analytics, RESTBase, Services, User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2194803 (mobrovac) [14:06:51] Analytics-Kanban: Update reportupdater's documentation - https://phabricator.wikimedia.org/T132326#2194875 (mforns) https://wikitech.wikimedia.org/wiki/Analytics/Reportupdater [14:09:43] Analytics, Community-Tech-Tool-Labs, Developer-Relations, MediaWiki-API, and 4 others: Determine which Action API parameters to whitelist/blacklist for action_param_hourly aggregate table - https://phabricator.wikimedia.org/T132283#2194878 (Anomie) I think a whitelist is going to be the safer appro... [15:17:21] Analytics-Kanban, Operations, ops-eqiad: Analytics1039 host showed high temperature alarms - https://phabricator.wikimedia.org/T132256#2195105 (elukey) @Southparkfan: thanks for the info! This is the first host that explicitly shows thermal errors, meanwhile the other one just rebooted for some reason... [15:36:36] (PS1) Mforns: Make code compatible with older versions of dateutil [analytics/reportupdater] - https://gerrit.wikimedia.org/r/282712 (https://phabricator.wikimedia.org/T131849) [15:36:38] HaeB: Hi sir [15:36:48] HaeB: I have some data for you :) [15:48:01] Analytics, Operations, Traffic: cronspam from cpXXXX hosts related to varnishkafka non existent processes - https://phabricator.wikimedia.org/T132346#2195218 (elukey) [15:52:15] Analytics, Operations, Traffic: cronspam from cpXXXX hosts related to varnishkafka non existent processes - https://phabricator.wikimedia.org/T132346#2195244 (elukey) [16:12:02] Analytics-Kanban, Patch-For-Review: Write hive code doing pageview data anonimisation with two tables {hawk} - https://phabricator.wikimedia.org/T118838#1810450 (JAllemandou) A non-prodictionized but working version of the code in the related patch. [16:24:50] Analytics-Kanban: Parse User-Agent strings with OS like "Windows 7" correctly into the user agent map {hawk} - https://phabricator.wikimedia.org/T127324#2039833 (JAllemandou) We decide to use UA-parser convention which is to separate major version version of windows (like Win7, winXP, win95 for instance) into... [16:25:04] Analytics-Kanban: Parse User-Agent strings with OS like "Windows 7" correctly into the user agent map {hawk} - https://phabricator.wikimedia.org/T127324#2195457 (JAllemandou) Open>declined [16:29:23] Analytics: Making tests environment for pageview API deployments - https://phabricator.wikimedia.org/T131773#2195483 (JAllemandou) p:Triage>High [16:34:11] Analytics: Productionitize druid - https://phabricator.wikimedia.org/T131974#2184741 (JAllemandou) p:Triage>High [16:37:59] Analytics-Kanban: Examine wikistats reports, make a summary of the most granular data needed that would serve all reports - https://phabricator.wikimedia.org/T131783#2195528 (JAllemandou) [16:40:08] Analytics-Kanban: Make legends on graphs better and more generic - https://phabricator.wikimedia.org/T129497#2195545 (JAllemandou) [16:51:00] Analytics-EventLogging, Analytics-Kanban: EventLogging dies when fetching a schema over HTTP that does not exist. {oryx} - https://phabricator.wikimedia.org/T124799#2195598 (mforns) a:mforns [17:05:55] a-team: I am about to log off, anything that you'd need from me before I leave? [17:06:10] Nothing on my side elukey :) [17:06:12] elukey, mmmmmm let meee think [17:06:14] Have a good evennin [17:06:16] xD no [17:06:24] good evening! [17:06:30] elukey: a plate of pastas from your region ? [17:06:39] ;) [17:06:46] gelatto? [17:06:58] hmmm, I wonder :) [17:07:02] * elukey is going to send https://en.wikipedia.org/wiki/Tortellini to joal [17:07:50] * joal 's mouth is watery [17:08:12] (PS1) Mforns: Change float to Decimal in dynamic_pivot.py [analytics/reportupdater-queries] - https://gerrit.wikimedia.org/r/282728 [17:08:18] joal: https://en.wikipedia.org/wiki/Lasagne :D [17:17:04] byeeeeee o/ [17:17:26] Ciao ! [17:20:39] hi all I just got back [17:20:48] hi madhuvishy! [17:20:56] ori: Hi! You're back :) [17:21:44] yep [17:27:18] ori: o/ in case you have time during the next days and you are interested https://gerrit.wikimedia.org/r/#/c/282652/ [17:29:37] * elukey logging off [17:33:30] elukey: woo, awesome [17:34:13] elukey: I'd love to review it, but I'm not sure when I'll have the time. I am drowning, at the moment :(. I'll try to get to it, but if time passes and I haven't reviewed it and it works, go for it. [17:34:22] (Draft1) Addshore: WIP DNM get dumps from archive.org if we want [analytics/wmde/toolkit-analyzer] - https://gerrit.wikimedia.org/r/282731 [18:17:50] joal: awesome! will take a look shortly (i'm officially on vacation today ;) [18:24:29] Analytics, Operations, Traffic: Generate a list of junk CN cookies being sent by clients - https://phabricator.wikimedia.org/T132374#2196113 (BBlack) [18:28:06] Analytics-Cluster, Operations: Migrate titanium to jessie (archiva.wikimedia.org upgrade) - https://phabricator.wikimedia.org/T123725#2196172 (Dzahn) [18:29:04] Analytics-Cluster, Operations: Migrate titanium to jessie (archiva.wikimedia.org upgrade) - https://phabricator.wikimedia.org/T123725#1936502 (Dzahn) https://wikitech.wikimedia.org/wiki/Analytics/Archiva @Analytics Does an upgrade of this server to jessie have blockers that are already known? [18:33:45] Enjoy your time HaeB, I won't erase the data ;) [18:37:56] Analytics, Operations: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#2196265 (Dzahn) p:Low>Normal could we raise the prio slightly to normal? are there other services here besides Apache? [18:46:08] joal: i'm getting this error: "SemanticException [Error 10041]: No partition predicate found for Alias "pageview_ios" Table "pageview_ios" ... [18:46:28] ...for this query: "hive (default)> SELECT year, month, day, CONCAT(year,"-",LPAD(month,2,"0"),"-",LPAD(day,2,"0")) as date, SUM(view_count) FROM joal.pageview_ios GROUP BY year, month, day ORDER BY year, month, day LIMIT 1000;" [18:46:59] ...anything wrong with the query? [18:59:52] HaeB: Since table is partitionned, you need to provide at least one partition info (like year > 0 for instance) [19:06:09] Analytics, Hovercards, Reading-Web-Sprint-71-m: Capture hovercards fetches as previews in analytics - https://phabricator.wikimedia.org/T129425#2196406 (GWicke) @phuedx, all accesses to RESTBase end points traverse text varnishes, which is where all logging happens the same way as for other requests.... [19:06:32] * joal is logging [19:06:34] off [19:44:02] joal: gotcha, forgot it needs to be in the WHERE clause [19:48:46] mforns: yt still? [20:02:57] how does one use https://vital-signs.wmflabs.org to see pageviews? [20:03:28] nuria_: hangout? [20:05:06] or, what's the correct graph for looking at pageviews? [20:07:12] HaeB: ^ [20:11:57] dbrant: vital signs reads off the pageview api now - so that would be correct [20:12:03] what are you looking for? [20:12:38] madhuvishy: just a general graph of pageviews, but i'm not seeing one [20:13:11] dbrant: https://vital-signs.wmflabs.org/#projects=all,eswiki,itwiki,enwiki,jawiki,dewiki,ruwiki,frwiki/metrics=Pageviews is the page that it launches which shows per project pageviews [20:13:50] madhuvishy: i don't see a graph when i click that link [20:13:56] dbrant: oh? [20:13:58] weird [20:14:08] do you have any errors on your console? [20:15:02] quite a few [20:16:01] the first one is "Failed to load resource: net::ERR_BLOCKED_BY_CLIENT" (https://piwik.wikimedia.org/piwik.js) [20:16:14] oh [20:17:02] followed by similar errors for the REST api requests [20:17:17] dbrant: can you try another browser/force cache clearing and reloading the page? i wonder if it's temporary [20:17:25] it loads fine for me [20:17:40] madhuvishy: lol it's my ad blocker! [20:17:52] ha ha [20:17:56] sorry for the nuisance! [20:18:02] i think there's some phab ticket about it [20:18:08] but we can't fix ad blockers :D [20:18:10] np! [20:18:39] if you are looking for a graph of per article pageviews - there's a tool here - tools.wmflabs.org/pageviews [20:27:30] Analytics, MediaWiki-extensions-CentralNotice, Operations, Traffic: Generate a list of junk CN cookies being sent by clients - https://phabricator.wikimedia.org/T132374#2196672 (AndyRussG) [20:27:41] Analytics, MediaWiki-extensions-CentralNotice, Operations, Traffic: Generate a list of junk CN cookies being sent by clients - https://phabricator.wikimedia.org/T132374#2196113 (AndyRussG) Thanks!! :) [22:06:10] Analytics-Kanban: limn-multimedia-data queries raising SQL syntax errors - https://phabricator.wikimedia.org/T132404#2197116 (mforns) [22:12:06] Analytics-Kanban: limn-multimedia-data queries raising SQL syntax errors - https://phabricator.wikimedia.org/T132404#2197161 (mforns) What happened is that the queries use the placeholder {wiki} in the FROM clause. And when using the `by_wiki` option, all wikis are instantiated into the placeholder, includin... [22:12:42] Analytics-Kanban: limn-multimedia-data queries raising SQL syntax errors - https://phabricator.wikimedia.org/T132404#2197166 (mforns) I added that potential solution to the Next features list in reportupdater documentation in Wikitech. [22:19:35] Analytics: Create data.wikimedia.org - https://phabricator.wikimedia.org/T132407#2197243 (Nuria) [23:42:59] It must be graph making season. Load average on stat1002 is over 30. :) [23:43:34] * James_F blames neilpquinn. :-) [23:47:49] * neilpquinn could not possibly comment. [23:48:13] bd808: where does that load avg number come from? [23:49:32] I'm reading it via `w`. In general load avg is the number of processes waiting for cpu time [23:50:58] bd808: cool, thanks! Don't think I have the access to do that :) [23:50:58] that box has 16 cores, so a load average of 30 means that the system is overloaded by ~90% [23:51:12] https://en.wikipedia.org/wiki/Load_%28computing%29#Unix-style_load_calculation [23:51:25] hah, wow, I can't say I'm surprised [23:53:05] * bd808 decides to do his data mining after dinner