[07:24:12] nuria: basically I want to download during runtime a webpage, and for that I need to use a proxy on the replicas, but when running the code locally on my computer for developing I of course can't connect with the proxy which only exists on the replicas. By now I first try to do connect with the proxy, and if it fails I try the other way, and were just wondering if there isn't a smarter way to do it. [08:10:01] jgonsior: hello! I'd add a parameter to your script called something like --proxy that force the use of a proxy only if set [08:18:56] Thanks, that will work for sure as well! [08:26:15] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun 2017): Find out (and fix) why we have a higher number of identity entries than before switching to new Bitergia DB scheme - https://phabricator.wikimedia.org/T168217#3377790 (10Aklapper) >>! In T168217#3363807, @Albertinisg wrote: > and "" is d... [09:18:34] 10Analytics-Kanban, 10Operations, 10Traffic: Artificial spike in offset of unique devices from November to February 6th on wikidata - https://phabricator.wikimedia.org/T165560#3377967 (10ArielGlenn) p:05Triage>03Normal [09:49:50] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services, 10scap2, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#3378114 (10hashar) Note: puppet is disabled on `deployment-aqs01` since June 8th though there is no reason given. The last Puppet run was at Thu Jun 8 13:29:4... [10:13:02] 10Analytics, 10Beta-Cluster-Infrastructure, 10Services, 10scap2, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#3378202 (10elukey) >>! In T116206#3378114, @hashar wrote: > Note: puppet is disabled on `deployment-aqs01` since June 8th though there is no reason given. > >... [10:22:30] (03PS3) 10Joal: Add two tables to sqoop on hadoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/360866 [10:29:50] 10Analytics-Kanban, 10Wikimedia-Stream: Port RCStream clients to EventStreams - https://phabricator.wikimedia.org/T156919#3378286 (10hashar) For Google, that probably comes from a time we provided them a stream of updates using OAI. There were several users of that service which we phased out end of 2013/earl... [10:40:09] (03PS4) 10Joal: Rename unique devices to per-project-family [analytics/refinery] - 10https://gerrit.wikimedia.org/r/360327 (https://phabricator.wikimedia.org/T168402) [10:54:21] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun 2017): Have "Last Attracted Developers" information for Gerrit (already exists for Git) - https://phabricator.wikimedia.org/T151161#2809332 (10Albertinisg) @Aklapper we've updated >>! In T151161#3176245, @Aklapper wrote: > Config: For Wikimed... [11:03:18] * elukey lunch! [11:05:41] Indeed, no comment :) https://twitter.com/samerbuna/status/785564422534209537 [11:40:46] (03CR) 10Joal: "daily job tested, monthly and druid not." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/360327 (https://phabricator.wikimedia.org/T168402) (owner: 10Joal) [11:41:37] joal: lol [12:16:54] (03PS1) 10Joal: Add Xml2Parquet Spark job [analytics/wikihadoop] - 10https://gerrit.wikimedia.org/r/361440 [12:19:38] (03PS11) 10Joal: Add new fields in mediawiki_history job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/359019 (https://phabricator.wikimedia.org/T161147) [12:36:28] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Improve purging for analytics-slave data on Eventlogging - https://phabricator.wikimedia.org/T156933#3378640 (10elukey) @Volans: still need to finalize some late requirements but we should be close, I'll ask a final review to Marcel and then I'll let yo... [13:13:13] for my dataviz lovers: https://uber.github.io/deck.gl [13:24:14] get started! npm install "aaaaargh my eyeeessss noooo" [13:36:40] fdans: the semantic autocomplete component isn't going to cut it, looking for alternatives, trying a few out like vue-typeahead [13:36:47] contemplating making my own [13:37:40] joal: the emoji MR is brilliant [14:24:17] milimetric: sorry I just saw this [14:24:24] sall good [14:24:25] what's missing from it? [14:24:27] man [14:24:28] :) [14:24:36] we can talk in the cve [14:24:47] milimetric: sure, let's go now [14:28:45] (03PS1) 10Joal: [WIP] Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) [14:28:54] 10Quarry: Slowdown of Quarry queries processing - https://phabricator.wikimedia.org/T168803#3379071 (10Mess) Have a look also on the Quarry maintanance request that was posted during the 2016 Community Wishlist Survey, where there are old reports similar to this one: https://meta.wikimedia.org/wiki/2016_Communit... [15:15:33] 10Analytics: Add normalized_host.project_family and deprecate and remove normalized_host.project_class - https://phabricator.wikimedia.org/T168874#3379221 (10JAllemandou) [15:18:41] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun 2017): Find out (and fix) why we have a higher number of identity entries than before switching to new Bitergia DB scheme - https://phabricator.wikimedia.org/T168217#3379239 (10Albertinisg) >! In T168217#3377790, @Aklapper wrote: > Do you plan... [15:21:11] 10Analytics, 10Community-Tech: Investigation: How can we improve the speed of the popular pages bot - https://phabricator.wikimedia.org/T164178#3379266 (10Milimetric) Let's make a task to output this data directly from the cluster so the bot doesn't have to do any work. [15:29:26] (03PS1) 10Joal: correct mediawiki history hive table scripts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/361471 [15:32:32] 10Analytics-Kanban: Collaborate with zero on asiacell report - https://phabricator.wikimedia.org/T161326#3379304 (10Nuria) a:03Nuria [15:37:48] 10Analytics, 10Operations, 10Traffic: Artificial spike in offset of unique devices from November to February 6th on wikidata - https://phabricator.wikimedia.org/T165560#3379313 (10Nuria) [15:38:43] Hi, im wondering if we can migrate some jobs from jdk 7 to jdk 8 please? [15:38:44] https://github.com/wikimedia/integration-config/blob/94f4a83548490f4015d1ad19ceaf79eff9304a22/jjb/analytics.yaml#L27 [15:39:04] Hi paladox - I'm sorry to say this in not possible nopw [15:39:11] oh [15:39:33] paladox: We're in plan to migrate hadoop to java 8, but it's huge in term of risk and infrastructure checks [15:39:43] ok [15:40:21] paladox: Please subscribe to T166248 to get updates - We'll get there at some point, but not soon [15:40:21] T166248: Upgrade Analytics Cluster to Java 8 - https://phabricator.wikimedia.org/T166248 [15:40:24] I think we may be able to do both java 8 and java 7. Jenins requires java 8 in the next update which has been released. Thus updating is on hold ow. [15:40:27] ow = now. [15:40:33] thanks [15:40:41] np paladox, sorry for the no-go [15:40:50] ok [15:41:46] 10Analytics, 10Analytics-Cluster: Upgrade Analytics Cluster to Java 8 - https://phabricator.wikimedia.org/T166248#3379338 (10Paladox) [15:41:57] 10Analytics, 10Analytics-Cluster: Upgrade Analytics Cluster to Java 8 - https://phabricator.wikimedia.org/T166248#3289857 (10Paladox) [15:46:36] 10Analytics: Productionize analysis of editcount vs per_user_revision_count - https://phabricator.wikimedia.org/T168648#3379394 (10Nuria) [15:48:52] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun 2017): Have "Last Attracted Developers" information for Gerrit (already exists for Git) - https://phabricator.wikimedia.org/T151161#3379400 (10Aklapper) >>! In T151161#3378329, @Albertinisg wrote: > Here are the new versions for Git and Gerrit:... [15:50:53] 10Analytics-Kanban: Add a job that regularly deletes druid webrequest deep-stored data - https://phabricator.wikimedia.org/T168614#3379403 (10Nuria) [15:53:45] joal i've added a subtask to jenkins upgrade task to say it's blocked by some tests that need to migrate to java 8 :) [15:54:17] Great - thanks paladox [15:54:27] your welcome :) [15:54:29] paladox: might push to move faster :) [15:54:37] thanks :) [15:55:41] The upgrade will fix some ssh problems (such as it using lower security it will soon be able to use higher security when we upgrade) :) [15:56:07] paladox: We actually also have tasks blocked by that upgrade, so yes, this hsould happen [15:56:14] 10Analytics, 10Analytics-Cluster: Perf test RAID vs JBOD with new hardware and kafka versions - https://phabricator.wikimedia.org/T168538#3379453 (10Nuria) We have been using for kafka single disks, as kafka knows where to put topic partititions. If a disk fails the broker needs to be shut down. We want to mea... [15:56:21] ok :) [15:58:02] 10Analytics: Data Lake queries abort with HDFS write fail - https://phabricator.wikimedia.org/T168497#3379467 (10Nuria) a:03JAllemandou [15:58:43] 10Analytics-Kanban: Data Lake queries abort with HDFS write fail - https://phabricator.wikimedia.org/T168497#3366377 (10Nuria) [15:59:45] 10Analytics-Kanban: Extraneous whitelist items for WikimediaBlogVisit schema - https://phabricator.wikimedia.org/T168475#3365522 (10Nuria) [16:00:46] 10Analytics, 10Analytics-Cluster: Make refinery drop data scripts email analytics-alerts if they fail - https://phabricator.wikimedia.org/T168415#3379474 (10Nuria) p:05Triage>03Unbreak! [16:01:22] 10Analytics-Cluster, 10Analytics-Kanban: Make refinery drop data scripts email analytics-alerts if they fail - https://phabricator.wikimedia.org/T168415#3379480 (10Nuria) [16:05:17] 10Analytics: Please update Tulu Language(tcy)in Wikipedia Statistics. - https://phabricator.wikimedia.org/T160630#3379494 (10Nuria) a:03Nuria [16:07:26] 10Analytics: Please update Tulu Language(tcy)in Wikipedia Statistics. - https://phabricator.wikimedia.org/T160630#3379509 (10Nuria) We will not be updating wikistats as we are about to replce that site in the next couple quarters but there are couple links here that you can use to look at pageviews for the site... [16:07:42] 10Analytics, 10Analytics-Wikistats: Please update Tulu Language(tcy)in Wikipedia Statistics. - https://phabricator.wikimedia.org/T160630#3379510 (10Nuria) [16:11:00] 10Analytics, 10Analytics-Dashiki, 10Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3379522 (10Nuria) I believe we abadoned patches right? Closing but please reopen if that is not the case. [16:11:09] 10Analytics, 10Analytics-Dashiki, 10Patch-For-Review: Create dashboard for upload wizard - https://phabricator.wikimedia.org/T159233#3379523 (10Nuria) 05Open>03declined [16:17:25] 10Analytics, 10Analytics-Cluster: Filter local IPs before checking for geo info - https://phabricator.wikimedia.org/T160822#3379551 (10Nuria) [16:17:27] 10Analytics: Incorporate data from the GeoIP2 ISP database to webrequest - https://phabricator.wikimedia.org/T167907#3379550 (10Nuria) [16:25:11] 10Analytics, 10Analytics-Cluster: Upgrade Analytics Cluster to Java 8 - https://phabricator.wikimedia.org/T166248#3379606 (10Nuria) @paladox: cluster upgrades and jenkins upgrades are really not related, removing subtask [16:25:28] nuria_: +1 thanks --^ [16:26:13] 10Analytics, 10Analytics-Cluster: Upgrade Analytics Cluster to Java 8 - https://phabricator.wikimedia.org/T166248#3379613 (10Nuria) [16:26:13] nuria_ That task is blocking the jenkins upgrade (unless i got the order wrong, got confused with parent and subtask) [16:26:22] paladox: it is unrelated [16:26:29] ok [16:26:37] paladox: we are talking "analytics cluster" [16:26:43] paladox: as in 50 hadoop nodes [16:26:49] ok [16:27:05] paladox: not jenkins nodes, but sure naming might be confusing [16:27:40] paladox: analytics code (that will execute tests on jenkins) runs fine on java8 [16:27:55] paladox: teh environment in which we run it is java7 however [16:28:00] paladox: hopefully that makes sense [16:28:05] ah, so we could possibly switch to java 8 on the jenkins tests? [16:28:18] paladox: ya, try them with java8 , tehy shoudl work [16:28:21] *they [16:28:27] Ok thanks :) [16:28:35] will do it now. [16:28:44] paladox: switching hadoop and teh millions of things that run there is another matter [16:28:51] ok [16:32:32] nuria_: https://gerrit.wikimedia.org/r/#/c/361482/ [16:34:17] 10Quarry: Slowdown of Quarry queries processing - https://phabricator.wikimedia.org/T168803#3379652 (10Mess) p:05Triage>03High [17:02:59] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421318 (10Mess) >>! In T139162#3288840, @Dvorapa wrote: > Is this still an issue? I haven't seen this issue for a while. Yes, it is indeed @Dvorapa. Check out my recent report here: https://phabricator.wikimedia.... [17:10:11] 10Analytics: Measure Community Backlog. - https://phabricator.wikimedia.org/T155497#2945157 (10Whatamidoing-WMF) On the general question of "What is the backlog?", the "WikiWork" concept might interest some people: https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2013-02-25/WikiProject_report The i... [17:16:21] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#3379891 (10Dvorapa) @Mess Recent queries list shows `queued` but the query is completed actually, you can see its real status if you click on some. The wrong status is tracked by T137517. But I think this issue sho... [17:23:42] * elukey off! [18:04:40] nuria_: I think we should keep an eye on the java8 jenkins thing [18:05:08] nuria_: Jenkins has refinery building for us, and if tests starts failing because of different versions we'll be in pain [18:26:21] (03PS2) 10Joal: [WIP] Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) [18:28:13] (03PS3) 10Joal: [WIP] Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) [18:31:14] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3380100 (10Quiddity) 05Open>03Resolved Marking as resolved. Seems to be fixed (per comments above). [18:32:38] !log help [18:32:39] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [18:33:29] milimetric: anything I can do in addition to the logging? [18:33:52] nono, just checking the docs [18:33:55] :) thx [18:34:03] :) [18:34:07] I just bounced quarry's runners and wanted to log it [18:34:56] okey [18:37:25] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10Patch-For-Review, 10Services (watching): Expose revision-create in EventStreams - https://phabricator.wikimedia.org/T167670#3380111 (10Halfak) @akosiaris, could you take a look at this? Once it's ready, we'll be able to simplify some precaching stuff for OR... [18:52:04] (03PS1) 10Joal: Update mediawiki history related tables [analytics/refinery] - 10https://gerrit.wikimedia.org/r/361500 (https://phabricator.wikimedia.org/T161147) [19:05:29] (03PS4) 10Joal: Add Clickstream builder spark job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/361459 (https://phabricator.wikimedia.org/T158972) [19:10:04] 10Analytics-Kanban: Data Lake queries abort with HDFS write fail - https://phabricator.wikimedia.org/T168497#3380183 (10JAllemandou) @Tbayer : Do you ming if I merge this task with T161147? The latter is the solution to the former. [19:31:48] halfak: Good morning sir [19:32:19] halfak: I added you to a CR for clickstream dataset generation in Spark [19:32:34] halfak: Code is currently running for enwiki, 2017-05 [19:32:37] ohhh! [19:32:43] Let's pull in shilad! [19:32:54] He's really excited about that and will be a good reviewer. [19:33:03] halfak: It'd be awesome if, when the run is finished, shilad could have a look at it and double check data sanity :) [19:33:09] halfak: awesome :) [19:42:44] halfak: Thanks a lot for the email translation :) [19:42:53] halfak: I'm **shy** [19:49:47] milimetric: I'm trying to access the EventLogging data on the analytics store (using the instructions on wikitech, but it's not working for me.... [19:49:49] kaldari@stat1003:~$ mysql --defaults-extra-file=/etc/mysql/conf.d/research-client.cnf --host dbstore1002.eqiad.wmnet [19:49:49] ERROR 1045 (28000): Access denied for user 'research'@'10.64.36.103' (using password: YES) [19:49:49] kaldari@stat1003:~$ mysql --defaults-extra-file=/etc/mysql/conf.d/research-client.cnf --host analytics-store.eqiad.wmnet [19:49:49] ERROR 1045 (28000): Access denied for user 'research'@'10.64.36.103' (using password: YES) [19:50:04] hm... [19:50:12] looking right now kaldari [19:50:15] thanks! [19:50:36] PROBLEM - Hadoop NodeManager on analytics1050 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:50:37] PROBLEM - Disk space on Hadoop worker on analytics1050 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:50:39] PROBLEM - Hadoop DataNode on analytics1050 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:51:53] kaldari: I can't see any difference, I got in with the same exact command I think: [19:51:53] mysql --defaults-file=/etc/mysql/conf.d/research-client.cnf --host analytics-store.eqiad.wmnet [19:52:04] weird [19:52:31] that worked [19:52:38] what? ... haha, diffing [19:53:36] oh, kaldari, yours uses the --defaults-file-extra [19:53:41] mine's just --defaults-file [19:53:45] yeah [19:53:46] I'm not sure why that would matter here [19:53:48] checking analytics1050.. [19:54:19] Thanks a lot elukey - was trying to get more info as well [19:54:33] but it works for me with defaults-extra-file too, maybe it was just rebooted kaldari [19:54:35] PROBLEM - YARN NodeManager Node-State on analytics1050 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:54:51] lemme know if it's still broken [19:54:59] or if I can help with a query [19:55:25] RECOVERY - YARN NodeManager Node-State on analytics1050 is OK: OK: YARN NodeManager analytics1050.eqiad.wmnet:8041 Node-State: RUNNING [19:55:35] RECOVERY - Hadoop NodeManager on analytics1050 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [19:55:36] RECOVERY - Disk space on Hadoop worker on analytics1050 is OK: DISK OK [19:55:38] RECOVERY - Hadoop DataNode on analytics1050 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [19:55:57] whaaat [19:56:19] haha, it's messing with you man [19:56:29] little disk dance [19:56:43] milimetric: For me it only works if I use --default-file, not --default-extra-file. I guess I'll change the documentation to say --default-file, since that works better :) [19:56:46] elukey: I promise I did nothing [19:56:57] kaldari: oh! you must have a .my.cnf file then [19:57:05] halfak: Does Silad have access to our cluster? [19:57:15] that's the rationale behind the -extra, that it doesn't override your normal .my.cnf [19:57:46] milimetric: Yes, I do have a .my.cnf. Any idea why that would keep it from working? [19:58:24] halfak: if so, file is ready :) [19:58:24] should I just delete that file you think? [19:58:38] halfak: if not I need to find a a way to give it to him [19:58:55] kaldari: no you don't have to, you can use --default-file instead, but that file must be overriding something that the research-client.cnf needs to set [19:59:05] I can't think what... maybe password? [19:59:13] got it. Thanks for the education :) [19:59:25] it seems that the raid controller froze on the host [19:59:33] if it's a file you need, you can rename it and use it explicitly in other connections, that'd be safest maybe [19:59:38] [Mon Jun 26 19:54:01 2017] megaraid_sas 0000:03:00.0: Reset successful for scsi0. [19:59:41] [Mon Jun 26 19:54:01 2017] megaraid_sas 0000:03:00.0: 2199 (2s/0x0020/CRIT) - Controller encountered a fatal error and was reset [20:03:16] elukey: is it in my dreams or our RAIDs controlers give us some trouble recently? [20:04:37] ok, done for today folks - see you tomorrow a-team [20:04:45] see you 'morrow [20:05:03] we also have the new kernel installed [20:05:12] RAID is a hard problem apparently, or so much fun that no engineers want to work on it ;) [20:06:49] weird thing is that we use the raid controller in a weird way [20:06:57] namely raid-0 with one disk [20:07:02] for each datanode partition [20:07:16] so except write cache we don't really use it [20:07:44] anyhow, will investigate an1050 tomorrow [20:07:46] seems working now [20:07:54] o/ [20:09:29] sorry for the late call, thanks Luca [21:07:38] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 5 others: Record an event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3380477 (10kaldari) @Ottomata: Once this starts collecting the page creation data in mySQL (hopefully sta... [21:10:49] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#3380482 (10Mess) @Dvorapa It's not true: I've clicked on [[ https://quarry.wmflabs.org/query/19775 | this query ]] and [[ https://quarry.wmflabs.org/query/19728 | this other one]] and their status is actually "que... [21:18:14] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 5 others: Record an event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3380507 (10Nuria) How to access data in MariaDB: https://wikitech.wikimedia.org/wiki/Analytics/Systems/E... [21:20:39] (03CR) 10Nuria: "Ok. Merging , task created to delete data from deep storage: https://phabricator.wikimedia.org/T168614" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) (owner: 10Joal) [21:21:04] joal: I always run tests on java8 [21:22:46] kaldari: i just updated docs on mariadb access and added those to ticket [21:23:01] thanks! [21:26:13] 10Analytics, 10Analytics-EventLogging, 10Contributors-Analysis, 10EventBus, and 5 others: Record an event every time a new content namespace page is created - https://phabricator.wikimedia.org/T150369#3380523 (10kaldari) 05Open>03Resolved Thanks Nuria. I think we can mark this resolved now! [21:27:19] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#3380530 (10Dvorapa) @Mess OK then. I experience same slow down as you describe. Sometimes even not very complicated queries take too much time. [21:29:46] (03CR) 10Nuria: "Before we merge this, can you remind me of the gotchas of rerunning these jobs?" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/355598 (https://phabricator.wikimedia.org/T166967) (owner: 10Joal)