[07:19:25] (03PS2) 10Amire80: Add new error types and abuse filter details printout [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/340982 (https://phabricator.wikimedia.org/T158834) [08:41:48] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#3074562 (10MoritzMuehlenhoff) @Milimetric : When new people should be added to the privileged LDAP groups ("wmf" for WMF staff or "nda" for volunteers/researchers/external contractors), please op... [08:48:06] Hi a-team [08:48:41] My IRC screen server got issues this weekend, I can't backlog the chan [08:49:39] elukey: would you mind letting me know if anything needs my attention? [08:51:04] joal: Morning! Nothing critical, the only thing that heppened was a disk issue with an1028 (there is an email about it) [08:51:47] elukey: I had seen that from alert email [08:52:26] I am wondering if it is worth or not to move the journal node away from an1028 [08:54:34] elukey: it probably depends on what the plans are in term of disk replacement [08:55:11] elukey: If disks can't be replaced soon, could be nice to move the journal out of an1028 and remove it entirely from the cluster for maintenance (maybe) [09:02:51] yep, I think that Chris will not be able to replace the disk before ~1 week [09:11:04] elukey: Then let's chane it ? could also be good to have the procedure for swapping from one node to another documented soewhere :) [09:14:12] it is already documented in https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#JournalNodes [09:14:17] but it requires restarting the namenodes [09:17:24] arf [09:21:07] elukey: Just need u to monitor closely, but shouldn't be too much of a difficulty? [09:25:54] joal: I am thinking if we need to do it or not.. atm only the journalnode daemon is active on the host (so HDFS datanote and yarn resource managers are down and puppet disabled). The journalnode daemons seems to work fine and afaics turning down the datanode "shields" the other daemons to be affected [09:26:09] (we could also think to start yarn again on it) [09:27:52] (I am not in slacker mode, only trying to mess with namenodes metadata as much as possible :D) [09:28:03] *not to mess [09:29:55] as far as I can see the journalnodes are writing to /var/lib/hadoop/journal/analytics-hadoop [09:30:00] (not on hdfs) [09:31:18] hello a-team :] [09:35:29] helloooooo mforns!! [09:35:32] how are things?? [09:35:38] hey elukey [09:35:47] everything's good :], you? [09:37:08] not bad :) [09:37:22] how is the new (Temporary) life? [09:38:18] good! quite hot, doing yoga and chilling out so far :] [09:38:31] nice :) [09:40:21] elukey, this is my ops week, is there any unplanned from last week concerning ops that I can help? Haven't read the emails yet [09:47:28] not really, but IIRC joal has been working on a lot of oozie jobs failing recently (after the cdh upgrade) [09:47:37] so he might need some info from you, but not sure [09:49:00] elukey, OK [09:55:17] Hi mforns ! [09:55:27] Glad to hear everything good for you :) [09:55:32] hello joal! :] [09:55:41] yea, all good! [09:55:44] mforns: oozie is back on track, so nothing really on my side [09:55:50] ok [09:56:00] what happened to it? [09:57:30] mforns: After CDH upgrade, some errors happened [09:57:57] 1- Unplanned error for all jobs about memory for oozie launchers (lead to restart all jobs) [09:58:20] I see [09:58:38] 2- Restart of all jobs lead to errors of long-not-restarted jobs havings references to suppressed jars [09:59:28] 3- Oozie + HiveContext in Spark seems buggy with the versions we have (ottomata commented on a ticket with cloudera° [09:59:52] So, given those errors: [09:59:55] oh [10:00:07] 1 - We modified all jobs to have the new oozie launcher conf param [10:00:48] 2 - We modified jobs with old jars to use new ones (1 non-backward compatible modif, the rest just version change) [10:00:56] aha [10:01:14] 3- We modified spark wikidata jobs to use sqlContext insteqd of hiveContext [10:01:33] And now oozie has not complained through the weekend (YES !!° [10:01:40] awesome [10:01:46] thanks for the update [10:02:04] :] [10:03:12] np mforns, it's been a bit busy last week :) [10:03:21] looks like it, eheheh [10:26:16] joal: I used a super hacky way to leave everything running on an1028 except the hdfs datanode, namely replacing /etc/init.d/hadoop-hdfs-datanode with exit 0 [10:26:28] in this way yarn and the journal node daemon will keep running [10:26:37] together with puppet [10:26:50] and the datanode will be left sleeping [10:27:01] this is a horrible version of the hammer [10:27:15] but eventually with Debian we'll have all the magic of "systemctl mask" [10:34:09] elukey: ok :) [10:35:31] joal: what do you think about reimaging an1040 to debian? [10:54:00] * elukey proceeds silently.. [10:54:48] elukey: sorry missed your ping [10:55:00] elukey: I don't really see how it affects ... [10:55:19] joal: all right I am going to do it! :) [10:55:24] so we'll have good data [10:55:35] maybe we could check if anything big is running on it [10:55:43] otherwise I'll stop the daemons and start the reimage [10:56:59] seems good [10:59:19] are there any hadoop instances in labs? [10:59:59] moritzm: yep on the analytics project (one is already running debian jessie, it is an hadoop worker node) [11:00:09] cdh5-3.eqiad.wmflabs [11:00:21] ok, does it currently use base::firewall? https://gerrit.wikimedia.org/r/341292 could potentially affect it, then [11:00:44] I mean, if it doesn't currently use it, then base::firewall would affect the labs instance as well [11:01:54] I am not aware of any use of base:firewall in labs for our things.. but the cluster is only for test purposes, nothing super important [11:02:02] so we can fix it later on if we see any issue [11:02:04] not a blocker [11:02:13] since the ferm rules for hadoop make extensive use of $ANALYTICS_NETWORKS to restrict access, which isn't available in labs [11:02:13] (not sure if I have answered your question) [11:02:36] yeah, you have, but then my patch can't be used as-is [11:03:29] I'll abandon, it's not really worth the effort to convert all the hadoop rules to be compliant with base::firewall in labs [11:04:23] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3070663 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1040.eqiad.... [11:07:22] elukey: just checked hadoop metrics on grafana: nodemanager heap is looking really better :) [11:07:48] elukey: however, hdfs files count is not - do you know if we've restarted files deletion cron ? [11:08:03] joal: good point, let me check [11:08:13] in the meantime, an1040 is being reimaged \o/ [11:09:20] Aawesome elukey :) [11:10:09] # Puppet Name: hdfs-balancer [11:10:10] 0 6 * * * /usr/local/bin/hdfs-balancer >> /var/log/hadoop-hdfs/balancer.log 2>& [11:10:15] mforns: I think webrequest cron deletion stopped was related to banners data recomputation, then blocked by data deletion strategy - any news on that? [11:10:35] is it the hdfs-balancer cron? [11:10:41] elukey: balancer and deletion are different :) [11:10:47] joal, I think Andrew switched it on again [11:10:55] yeah I suspected that [11:12:06] mforns: ok, let's double check elukey, but maybe file growthv is just natural given jobs logs [11:12:14] joal, I don't think that banner data deletion/sanitizing strategy needs webrequest data no? [11:12:33] joal: do you know where the cron should be running [11:12:34] ? [11:12:54] mforns: I think the reason for which we stopped deletion was to allow recomputation of december data using deployed oozie jobs [11:13:02] But, those have not been started mforns [11:13:12] elukey: trying to remeber [11:14:48] elukey: I don't remeber, and I have not found it online [11:14:55] elukey: Let's wait for ottomata [11:15:18] mforns: do those oozie stuff ring bells, or not really? [11:15:18] joal, yes, but that is done, and even we have no sanitizing strategy yet, we should reset 60-day-purging for webrequest [11:15:18] joal, but I'm almost sure Andrew did that already [11:16:01] mfournier: you said that is done - what is done? [11:16:17] mforns: --^ sorry [11:16:37] mfournier: My apologizes, was not adressed to you (tab error) [11:17:00] joal, reloading of december with new data format is done [11:17:13] joal: I didn't find it on puppet, not sure what it should do.. I thought it was part of the hdfs balancer script [11:17:17] joal, and also I think Andrew did reset the 60 day limit for webrequest purging [11:18:04] mforns: ok [11:18:35] mforns: should we start regular loading for banners then? [11:18:56] elukey: mforns says cleanup has been started back by ottomata [11:19:06] joal, sure, I thought we already did... sorry my bad [11:19:17] mmmmmmmmmmmmmmmm [11:19:22] :D [11:19:29] * mforns searches for gerrit change [11:19:37] np mforns, I thought we did as well (realized that when fixing the mess last wekk) [11:20:29] mforns: The info missing for me to stqrt them was: what should we pick as start date? [11:20:43] joal, I see [11:20:52] looking [11:21:54] joal, for the daily job, Feb 1st [11:22:46] joal, for the monthly, I don't remember if january was already compacted using the monthly job or not... jan has full data, but maybe we could start the monthly job at Jan 1st to be sure [11:23:13] mforns: feb has full data as well: realtime provides it [11:23:54] joal, if I look at pivot, I can see a hole, and anyway, the daily job adds a couple fields no? [11:23:59] mforns: But, when looking at pivot, it seems to be missing fields that are computed by oozie [11:24:31] Ah mforns ! [11:24:35] joal, from feb 1st to feb 7th, I see a hole, am I looking at it right> [11:24:36] ? [11:24:46] mforns: I have not restarted my RT job after cluster upĝrade !!! [11:25:12] Ah! OK OK [11:25:31] then yes, monthly job: 1st Jan; daily job: 1st Feb [11:25:44] joal, no wait! [11:25:55] we can let the monthy job calculate feb as well, sorry [11:26:28] monthly job: 1st Jan; daily job: 1st Mar [11:26:41] no? [11:27:40] mforns: I don't know :) [11:28:14] mforns: restarting realtime job [11:28:18] joal, ok [11:28:24] mforns: This needs to run anyway [11:28:45] mforns: on monthly / daily jobs, best could be to check segments in druid [11:29:05] aha [11:30:00] joal, actually, I remember having run monthly for january [11:30:47] joal, elukey: that's Andrew's patch resetting webrequest purge to 62 days: https://gerrit.wikimedia.org/r/#/c/336458/ [11:31:59] awesome mforns, thanks for checking [11:32:13] np :] [11:32:51] and joal, I'm totally sure that the monthly job was run for jan, we can start monthly at 1st Feb and daily 1st Mar [11:33:53] joal, do you want me to change the default properties in the oozie job? or will we -D override those in the oozie call? [11:35:13] mforns: just checked data on druid --> indeed jan got compacted monthly [11:35:22] aha [11:35:35] mforns: I'm not sure which properties you're talking about :) [11:35:45] joal, xD [11:35:59] the properties file for the oozie job has the start and stop parameters [11:36:14] do you want me to change them? to make the start dates "official"? [11:36:26] although, the data starts at 2016-11-28... [11:36:28] mforns: no need - We always use -D to override when we restart [11:36:33] ok ok [11:37:45] joal: ahh the script runs on an1027 [11:37:53] nice to know [11:37:56] afaics it is enabled [11:41:12] Debian GNU/Linux 8 analytics1040 ttyS1 [11:42:07] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3075096 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1040.eqiad.wmnet'] ``` and were **ALL** successful. [11:44:21] what the hell... all the hadoop partitions are not mounted? [11:44:24] on an1040 [11:44:26] sigh [11:44:30] it was too good to be true [11:44:32] elukey: then as I said, file growth must be related to new jobs [11:45:00] elukey: unfortunately, I was expecting at least some things to break [11:45:18] mforns: D you mind restarting the oozie jobs for banners? [11:45:34] joal, sure will do [11:45:38] ahhhh no it is me the problem [11:45:40] as always [11:45:45] the partition creation is manual! [11:45:49] elukey: it is usual :) [11:46:01] elukey: I'm my own problem most of the time ;) [11:46:12] Guys, away for some time to lunch [11:46:16] see you in a biy [12:05:11] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3075124 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by elukey on neodymium.eqiad.wmnet for hosts: ``` ['analytics1040.eqiad.... [12:05:21] of course I messed up with the partitions :D [12:05:33] re-installing [12:11:37] and I confirmed that the partman recipe doesn't work as expected [12:11:49] it stops before saying "Yes" to the new partitions [12:11:49] sigh [12:23:07] * elukey lunch! [12:38:52] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Reimage a Trusty Hadoop worker to Debian jessie - https://phabricator.wikimedia.org/T159530#3075210 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['analytics1040.eqiad.wmnet'] ``` and were **ALL** successful. [12:59:01] (03CR) 10Mforns: "Hey, I can not see the changes that deploy to production by default. Am I missing something?" (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/340375 (owner: 10Milimetric) [13:15:01] hiiii, sorry, been all day with my irc window closed >.< [13:31:31] (03Draft1) 10Amire80: Add Google Spreadsheet editor [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/341315 [13:32:05] (03Abandoned) 10Amire80: Add Google Spreadsheet editor [analytics/limn-language-data] - 10https://gerrit.wikimedia.org/r/341315 (owner: 10Amire80) [13:51:58] hi fdans :] [13:54:22] (03CR) 10Milimetric: "mforns: I just removed -test from the hostname of each dashboard in config.yaml" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/340375 (owner: 10Milimetric) [13:55:22] milimetric, oh yea, I was blind indeed [13:56:32] cool, sorry I merged before you looked, but the dashiki extension merged so now we have nice pages on meta: https://meta.wikimedia.org/wiki/Config:Dashiki:VitalSigns [13:56:55] mforns: interested in your thoughts about the other configs, like CategorizedMetrics, that are still left in raw-json [13:57:15] I made this task: https://phabricator.wikimedia.org/T159269 [13:57:21] milimetric, awesome! [13:57:47] * mforns looks [13:59:13] so now I am really puzzled [13:59:42] the partman recipe to set up our worker nodes creates a swap lvm partition of 240GB :D [14:00:36] 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, 10Research-management: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#2734568 (10Ladsgroup) Regarding GPU options. I just want to note that their drivers are propriety software and not open source (or partiall... [14:00:49] milimetric, but the patch is not merged yet no? [14:01:05] oh, you deployed without merging? [14:01:31] I need master ottomata :D [14:02:24] ops people are so cryptic... [14:02:38] come oooonnn [14:02:50] xD [14:02:50] :) [14:03:49] basically I am trying to figure out why partman (a lovely thing that configures for you disk setups with a very understandable syntax) is creating partitions in a weird way [14:03:49] mforns: oh! oops, yeah, deployed without merging [14:03:57] ok ok [14:04:00] so... [14:04:10] ottomata: hhhhhhhhhhhhhhhhiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii [14:04:10] (03CR) 10Mforns: [V: 032 C: 032] Clean up Config [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/340375 (owner: 10Milimetric) [14:04:24] hiiiii [14:04:40] whenever you are coffeinated and ready to go I'd need to ask you some questions [14:06:56] k eatin some cereal, checking emails... [14:06:58] :) [14:08:14] ottomata: does it mean that I can start? :D [14:09:07] haha, uhh, give me a few! [14:09:07] 10Analytics, 10Analytics-Dashiki: Clean up remaining Dashiki configs on meta - https://phabricator.wikimedia.org/T159269#3062327 (10mforns) @Milimetric > Available projects should be phased out (it already partially is) in favor of just parsing the site matrix. Totally agree. > Annotations, out-of-service,... [14:09:14] milimetric, ^ [14:09:30] ottomata: ahhh okok! even one hour, not really urgent [14:13:17] 10Analytics, 10Analytics-Dashiki: Clean up remaining Dashiki configs on meta - https://phabricator.wikimedia.org/T159269#3075565 (10Milimetric) >> Annotations, out-of-service, and metrics should be moved to a new sub-domain, maybe Config:DashikiMeta:. > Agree as well, but... Does this mean that we have to go t... [14:17:51] 06Analytics-Kanban, 15User-Elukey: Ongoing: Give me permissions in LDAP - https://phabricator.wikimedia.org/T150790#3075598 (10Milimetric) Could we just change the phrasing of this task to say that? So the description would be: To use Pivot, Piwik, and other Analytics tools that require an LDAP login, please... [14:29:42] going to start writing.. [14:29:52] so today I reimaged analytics1040 to debian [14:30:26] all good but when I tried to create the journalnode partition I got an error that the LVM VGS was full, no more space to allocate the 10G partition [14:30:58] I checked and the partman recipe created, as requested, a 30GB root partition and the rest was allocated for SWAP [14:31:08] in Trusty nodes, the swap is 1G [14:31:18] on analytics1040, it is 230GB [14:31:42] so I checked and we have tons of space in the VGS on a regular worker ndoe [14:31:45] *node [14:31:52] root@analytics1041:/home/elukey# vgs -o +vg_free_count,vg_extent_count VG #PV #LV #SN Attr VSize VFree Free #Ext analytics1041-vg 1 3 0 wz--n- 232.09g 193.22g 49465 59415 [14:31:59] horrible format sigh [14:32:08] root@analytics1041:/home/elukey# vgs -o +vg_free_count,vg_extent_count [14:32:16] VG #PV #LV #SN Attr VSize VFree Free #Ext [14:32:22] analytics1041-vg 1 3 0 wz--n- 232.09g 193.22g 49465 59415 [14:32:27] this --^ is a trusty node [14:32:37] that shows 193GB of free space [14:32:41] am I reading it wrong? [14:33:24] I checked pvdisplay and lvdisplay, it seems that we have space remaining in the flex bays [14:34:08] I also filed https://gerrit.wikimedia.org/r/#/c/341318/3/modules/install_server/files/autoinstall/partman/analytics-flex.cfg becuase the Debian installer stops before creating the partitions [14:34:12] asking for a Yes or No [14:52:56] (03PS2) 10Joal: [WIP] Add oozie jobs for mw history denormalized [analytics/refinery] - 10https://gerrit.wikimedia.org/r/341030 [14:54:20] in the meantime, I merged https://gerrit.wikimedia.org/r/#/c/341318/ [14:55:19] (03PS20) 10Joal: Add mediawiki history spark jobs to refinery-job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/325312 (https://phabricator.wikimedia.org/T141548) [14:59:59] milimetric: Hello! I am reading your comments for the LDAP access for Pivot, but I am wondering why the one-task-per-user solution is not ok for you (or better you'd prefer another way) [15:01:46] elukey: one-task-per-user is great, it's just that if I point someone to a wiki page that describes that, they almost always come back with questions. If I point them at a task in phabricator, they tend to follow the directions there better. And it's easy for me to find the task and link it [15:01:52] elukey: ok! [15:02:08] so, iirc, i could never get partman to work properly for JBOD for hadoop [15:02:20] especially with special cases for the flex bays [15:02:22] I totally agree with you and moritz that you shouldn't handle those requests as comments, that's what I was saying, did that come across different? [15:02:46] as for free space in flex bay vgs [15:02:59] that sounds fine i think. i don't fully remember, but i think i would have done something like that [15:03:07] if there wasn't a reason to allocate all of the space [15:03:17] i like to leave some in the VGs so that we can allocate as necessary later [15:03:21] its easier to add space than remove it [15:04:10] ahhh okok [15:04:27] makes sense, maybe I'll document it so everybody will know it :) [15:04:32] it was a bit strange [15:04:47] milimetric: nono sorry I thought you wanted to handle the future cases in a different way, nevermind! [15:04:56] also, elukey, its good that you are testing this partman repartitioning stuff for jessie...but i would think hopefully we wouldn't have to wipe the non root partitions [15:05:59] ottomata: yep yep this would be the second step, I wanted to make sure that the recipe was working fine [15:06:29] ok cool [15:07:17] ottomata: weird thing is, the swap partition part in the partman recipe seems to get the whole space available [15:07:21] rather than 1GB [15:07:37] there must be a little/subtle issue with partman [15:07:42] will keep investigating, thanks [15:07:43] hm, yeah that's weird [15:07:54] i know how much you love working with partman [15:07:56] go get em! [15:10:36] ottomata: last thing - today I used a horrible trick to force only the hdfs datanode daemon to stop on an1028 [15:10:49] namely replacing all the content of its init.d file with exit 0 [15:11:04] I realized that yarn and journal node daemons were not affected by the disk failure [15:11:09] so I haven't moved the journal node [15:11:17] but we can do it [15:12:53] hmm [15:16:02] atm an1028 works fine with puppet enabled etc.. [15:16:08] it was basically a "systemctl mask" [15:16:10] horrible [15:19:49] hm [15:19:58] so chris is out all this week? [15:21:14] IIRC yes, but today he might be in the DC [15:21:15] not sure [15:21:33] oh just today? ideally we can just swap the disk and auto rebuild [15:21:44] buuuut, ja if a week, it might be good to actively move the journalnode [15:21:49] and, hey, maybe good practice for us anyway? [15:22:23] yes it might but I saw the procedure and I wanted to ask you first, since I don't like to mess with the Master nodes :) [15:22:30] maybe we could do it tomorrow together? [15:22:37] if nothing changes [15:23:06] (we also need to apply my script to generate the fstab before doing anything) [15:25:17] k ya let's do tomorrow [15:25:25] we can practice in labs, ja? [15:25:30] sure [15:25:31] k [15:25:32] another thing [15:25:39] do we need a 1g swap partition? [15:25:42] on worker nodes [15:26:20] plus a root bigger than 30GB would be good [15:26:22] like 60 [15:26:31] we have space in there :) [15:26:52] i like bigger roots too, but i was told by other ops folks: what's the point, you should have good log rotation, we default to 30ish G roots blabla [15:27:09] 1g swap? no idea really. [15:27:16] i think swap probably doesn't really help us much [15:27:23] I'd vote to nuke it [15:27:27] +1 [15:27:28] go for 60GB of root [15:27:37] i'm into it [15:27:40] we do what we want. [15:27:43] :) [15:27:46] yesss [15:28:03] and also maybe 20GB journal by default? Rather than adding it via script [15:33:36] if you have learned the proper partman incantation, i bow to your wisdom :) [15:34:35] 10Analytics-Tech-community-metrics: Updated data in mediawiki-identities DB not deployed onto wikimedia.biterg.io? - https://phabricator.wikimedia.org/T157898#3075984 (10Aklapper) Wondering if this is related (I should check once this task is resolved): The account https://phabricator.wikimedia.org/p/EddieGP/ ha... [15:42:57] ottomata: the mountpoint might be weird, since it is /var/etc.. [15:43:24] ho right [15:43:30] those dirs are created by puppet [15:43:36] /var/lib/hadoop/journal [15:43:39] yep.. [15:43:42] /var/lib/hadoop might be created by the package [15:43:45] but in either case, ja [15:43:45] hm [15:43:54] woudl it be possible to create the partition, but not mount it? [15:43:55] with partman? [15:44:04] the main weird thing is that partman seems to allocate all the free space rather than leaving something free [15:44:13] not sure about it [15:48:32] ya tha'ts pretty weird [15:48:37] in the swap partitoin, eh? [15:53:36] yes exactly.. [15:53:42] not sure if it is a new behavior [15:53:52] or an error in the partman recipe [15:53:57] but it worked in the past right? [15:54:17] I mean, the current nodes have been installed with the current analytics-flex recipe right? [15:59:32] 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, 10Research-management: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#3076068 (10Ottomata) Q: Does T159165 mean that we no longer need to get a new stat box with a GPU? Or is this ticket still valid? I'm ab... [15:59:41] yeah [15:59:48] prettys sure that worked in the past [16:01:17] elukey, milimetric : standdupp [16:01:37] (03CR) 10Nuria: Clean up Config (031 comment) [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/340375 (owner: 10Milimetric) [16:02:21] joal: standduppp [16:03:18] trying to join [16:03:36] rebooting [16:13:23] 10Analytics-EventLogging, 06Analytics-Kanban, 13Patch-For-Review: Change userAgent field to user_agent_map in EventCapsule - https://phabricator.wikimedia.org/T153207#3076096 (10Nuria) a:05Nuria>03fdans [16:29:33] ottomata: Was asking if you needed me for event [16:29:39] bus meeting [16:30:06] oh [16:30:10] naw joal you can skip [16:30:19] k thanks :) [16:30:30] with a bad connection, what point anyway ottomata ;) [16:31:18] :) [16:32:28] Bye team, sorry for bad connection [16:36:20] ottomata: https://lwn.net/Articles/690079/ - really nice [16:36:44] it is a very clear view of swap, I was maybe too aggressive in removing it.. not sure [16:37:24] 1G seems really low, but maybe something more could be usful [16:37:26] *useful [16:37:36] but it is always a tradeoff [16:45:22] mforns: still with us? [16:45:30] joal, yes [16:45:39] s'up [16:45:50] 10Analytics, 10Analytics-Cluster, 06Operations, 06Research-and-Data, 10Research-management: GPU upgrade for stats machine - https://phabricator.wikimedia.org/T148843#3076221 (10Halfak) I think that having a GPU in a stats machine for modeling work will be critical for the research team and any other mode... [16:45:52] :] [16:45:54] heyq, is it on purpose you've launched banners jobs in default queue? [16:46:09] joal, O.o [16:46:23] oops [16:46:32] ok, let me fix that [16:47:43] mforns: no big deal, don't kill currently running one maybe? [16:48:36] ottomata: I think that in Debian partman assigns all the free space left to the last LVM volume :( [16:48:44] and there seems to be no option to avoid it [16:49:23] 10Analytics, 10Recommendation-API: productionize recommendation vectors - https://phabricator.wikimedia.org/T158973#3076230 (10Fjalapeno) [16:54:59] mforns: it also seems you've relaunched jobs for January, no? [16:55:49] joal, january? I hope not... [16:56:11] hue says 2017-2 [16:56:25] mforns: it's weird, look at the title of the currently running job in yarn [16:57:17] joal, it says 2017-2 no? [16:57:38] oozie:launcher:T=hive:W=banner_activity-druid-monthly-wf-2017-2 [16:58:59] mforns: was looking at druid job, but it looks my eyes have fooled me [16:59:47] mforns: sorry for making this fals alarm - I'm gonna stop doing operations for today I think :) [17:00:04] joal, don't worry, thanks for spotting the queue_name issue [17:01:05] will wait for the daily job to finish and restart the daily coord [17:03:17] elukey: :/ [17:03:32] i guess if we do 60G for root, and just let it to do the rest for journalnode partition [17:03:34] it'll be ok [17:06:16] ottomata: I was more for the opposite way around, namely root with all the remaining space, but maybe if we find a way to set up the journalnode partion it might be an option [17:06:26] and then we can add a step to resize it after installing [17:07:17] elukey: , or, just add a fake space partitoin [17:07:20] that we won't ever use [17:07:26] and just can delete it if/when we need the space [17:07:47] yeah [17:08:20] the only weird thing is that it might require a mountpoint [17:08:30] I saw somebody on the internetz using /tmp/hack [17:20:14] https://www.mediawiki.org/wiki/Architecture_committee/2017-03-01 - last news is really interesting :) [17:30:38] elukey: looking [17:31:04] elukey: ahhh yess, for got to share that [17:31:17] elukey: because we have not had staff [17:31:36] it was super sneaky but I am really happy about it! [17:31:50] I think we need to say thank you to Victoria :) [17:33:32] elukey: sorry, i should have shared that earlier as it s been on teh works for a while [17:37:05] nuria: nono not your fault! This news should have needed a wmf-all email probably :) [17:58:38] elukey: i think the $$$ need to be moved before announcing how big is the team, that is wip [18:00:01] ahhh [18:03:50] 10Analytics: Refactor monthly banner oozie job to use already indexed daily data - https://phabricator.wikimedia.org/T159727#3076363 (10JAllemandou) [18:04:38] ottomata: Should we in meeting? [18:04:46] Or am I in an alternate place? [18:18:06] 10Analytics, 10EventBus, 06Services (watching): EventBus logs don't show up in logstash - https://phabricator.wikimedia.org/T153029#3076457 (10mobrovac) [18:25:31] * elukey off! [18:25:47] ottomata: I left an1040 with hadoop daemons masked and icinga silenced [18:25:54] will restart tomorrow on partman [18:27:04] oook! [18:30:19] 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 10Reading Epics (Trending Edits), and 3 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#3076581 (10Jdlrobson) [18:40:39] ashgrigas: is it ok if I post your dropbox links publicly on the repository where we're doing the prototype? [18:40:50] I mean, they're public here too, but just making sure [18:42:44] I can remove it if you want: https://github.com/milimetric/wikistats-prototype [18:46:04] milimetric sure [18:46:11] thats fine! [18:46:25] 10Analytics, 10ChangeProp, 06Operations, 10Reading-Web-Trending-Service, 06Services (watching): Build and Install librdkafka 0.9.4 on SCB - https://phabricator.wikimedia.org/T159379#3065770 (10Ottomata) [18:52:14] 10Analytics, 10EventBus, 10Reading-Web-Trending-Service, 10Reading Epics (Trending Edits), and 3 others: Compute the trending articles over a period of 24h rather than 1h - https://phabricator.wikimedia.org/T156411#3076688 (10mobrovac) Will try to get it scheduled for this week. Apologies for the delay. [18:55:22] 10Analytics, 06Operations, 06Performance-Team: Update jq to v1.4.0 or higher - https://phabricator.wikimedia.org/T159392#3076697 (10Ottomata) p:05Triage>03Low a:03Ottomata I'll take this on, low priority though. Remind me about it if you get fidgety! :) [18:56:59] 10Analytics, 06Operations, 06Performance-Team: Update jq to v1.4.0 or higher - https://phabricator.wikimedia.org/T159392#3066309 (10MoritzMuehlenhoff) jessie has jq 1.4, so this would also be fixed once stat1002 is migrated to jessie. [18:57:45] 10Analytics, 10ChangeProp, 06Operations, 10Reading-Web-Trending-Service, 06Services (watching): Build and Install librdkafka 0.9.4 on SCB - https://phabricator.wikimedia.org/T159379#3076717 (10Pchelolo) After some testing of driver-librdkafka compatibility, here's the deal: 1. Currently we are using `nod... [19:07:41] 10Analytics, 06Operations, 06Performance-Team: Update jq to v1.4.0 or higher - https://phabricator.wikimedia.org/T159392#3076750 (10Krinkle) @MoritzMuehlenhoff Thanks. Is there a ticket for that? I've transferred my data to terbium for post-processing for the time being because the python/ua-parser package... [19:41:27] 10Analytics, 10Analytics-Cluster: Enable hyperthreading on analytics100[12] - https://phabricator.wikimedia.org/T159742#3076950 (10Ottomata) [20:08:25] Is this the right place to request some eventlogging data? Pertaining to https://meta.wikimedia.org/wiki/Schema:CookieBlock Revision ID 16241436 [20:09:38] hi Niharika, I can help [20:09:52] Hey milimetric. [20:09:55] Awesome. [20:09:57] are you looking to query the data ad-hoc or make reports? [20:10:20] milimetric: Just querying ad-hoc. [20:10:30] ok, Niharika do you have access to stat1003.eqiad.wmnet? [20:10:40] EventLogging data is replicated there [20:10:48] Let me see. [20:11:15] milimetric: I don't think so. [20:11:26] Asks for a password if I try to ssh. [20:11:46] ok, and is your ssh setup so you can access other machines on eqiad? [20:12:04] like, using the bastion and forwarding your key and all that? [20:12:22] milimetric: Yep. [20:13:56] ok, Niharika, then you'll need to request access. So you file a task with Ops-Access-Requests and you ask for the groups "researchers and statistics-users" [20:14:01] more info can be found here: https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Access_Groups [20:14:31] you'll have to wait though because they have a waiting period on all access requests. In the meantime, is there anything you need urgently Niharika ? [20:15:50] there are 681 records in that table (the version of CookieBlock you need), so I can check whatever you're looking for pretty easily probably [20:22:07] Niharika: did I lose you? [20:22:31] 10Analytics, 10EventBus, 06Services (watching): EventBus logs don't show up in logstash - https://phabricator.wikimedia.org/T153029#3077255 (10Pchelolo) The https://phabricator.wikimedia.org/T150106#2777178 should've been resolved by https://github.com/wikimedia/change-propagation/pull/133 [20:23:00] milimetric: Sorry, I'll Be right back. [20:23:12] np, just making sure I wasn't being confusing [20:31:05] milimetric: Sorry again. I'm filing a request now. It's not urgent, so I can wait till I get access. Thanks! [20:32:13] k, Niharika when you get access what you'll need is to mysql into analytics-store.eqiad.wmnet and use the "log" database, and select count(*) from CookieBlock_16241436; [20:32:27] Noted. [20:45:59] ottomata (or joal) yt? [20:46:12] ya [20:46:14] nuria: hey [20:46:32] ottomata: one question that might be very easy [20:46:43] ottomata: [20:46:52] I am trying to insert data into a table on my db [20:46:56] https://www.irccloud.com/pastebin/P5pfPULR/ [20:47:29] ottomata: [20:47:31] like: [20:47:32] hive -f ./insert_test.hql -d destination_table=nuria.last_access_uniques_daily_asiacell -d year=2017 -d month=01 -d day=10 [20:47:48] ottomata: so destination table is nuria.last_access_uniques_daily_asiacell [20:48:22] ottomata: but i get an error about "moving" files [20:48:24] ottomata: Failed with exception Unable to move source hdfs://analytics-hadoop/tmp/hive-staging_hive_2017-03-06_20-43-00_938_6844714293285814446-1/-ext-10000 to destination hdfs://analytics-hadoop/wmf/data/wmf/last_access_uniques/daily/year=2017/month=01/day=10 [20:49:27] nuria: do show create table [20:49:28] on your table [20:49:32] what is the 'location'? [20:49:39] it looksl ike it is pointing into /wmf/data/wmf/last_access_uniques [20:49:46] which is the prod table, which your user does not have perms to write to [20:50:06] looks like you copy/pasted the create table statement, but didn't alter the location [20:50:38] ottomata: [20:50:42] https://www.irccloud.com/pastebin/gB3Up40M/ [20:50:52] ottomata: argh, it is!!!!! [20:50:56] :) [20:51:08] ottomata: thank youuuu [20:53:52] yw [20:54:56] 10Analytics, 10ChangeProp, 06Operations, 10Reading-Web-Trending-Service, 06Services (watching): Build and Install librdkafka 0.9.4 on SCB - https://phabricator.wikimedia.org/T159379#3065770 (10mobrovac) There's no need to have downtime at all for the upgrade - we have multiple hosts for these services an... [20:55:11] 10Analytics-Cluster, 06Analytics-Kanban, 06Operations, 13Patch-For-Review: Move cloudera packages to a separate archive section - https://phabricator.wikimedia.org/T155726#3077357 (10Ottomata) Ha! OOPS I knew I would make a mistake when cleaning up old packages! I accidentally removed almost all CDH pack... [20:55:49] 10Analytics, 10ChangeProp, 06Operations, 10Reading-Web-Trending-Service, 06Services (watching): Build and Install librdkafka 0.9.4 on SCB - https://phabricator.wikimedia.org/T159379#3077359 (10Pchelolo) >>! In T159379#3077355, @mobrovac wrote: > There's no need to have downtime at all for the upgrade - w... [20:56:12] 10Analytics, 10ChangeProp, 06Operations, 10Reading-Web-Trending-Service, 06Services (watching): Build and Install librdkafka 0.9.4 on SCB - https://phabricator.wikimedia.org/T159379#3077373 (10mobrovac) [22:53:27] 10Analytics, 10ChangeProp, 06Operations, 10Reading-Web-Trending-Service, 06Services (watching): Build and Install librdkafka 0.9.4 on SCB - https://phabricator.wikimedia.org/T159379#3077953 (10Ottomata) By adding a new version of librdkafka to our apt repo, it has the chance that it might also be install...