[00:00:56] 10Analytics-Cluster, 10Analytics-Kanban, 10Language-Team, 10MediaWiki-extensions-UniversalLanguageSelector, and 3 others: Migrate table creation query to oozie for interlanguage links - https://phabricator.wikimedia.org/T170764#3441928 (10Milimetric) Ready to review, @Amire80. Please take a look at the ta... [05:44:27] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659536 (10Marostegui) >>! In T153033#3658843, @demon wrote: >>>! In T153033#3656161, @Marostegui wrote: >> Just to be clear, you are talking about dbstore1002/db1047? >> We also have to keep in mind that there a... [07:16:02] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659635 (10Marostegui) I have checked the tables across the masters: s1: has data on `enwiki` s2: has data only on: `nlwiki` s3: has data only on: ``` frwikisource incubatorwiki itwikivoyage sewikimedia tawiki... [07:48:10] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659653 (10demon) Testwiki we can drop for sure. So that just leaves 7 total wikis with viable data. Farrrrrrrr better. [07:52:46] 10Analytics, 10DBA: Drop MoodBar tables from all wikis - https://phabricator.wikimedia.org/T153033#3659668 (10Marostegui) Thanks @demon! I will exclude testwiki from the list of wikis the tables need to be imported from [08:34:57] 10Analytics, 10User-Elukey: Add the prometheus jmx exporter to all the Hadoop daemons - https://phabricator.wikimedia.org/T177458#3659718 (10elukey) [08:40:38] 10Analytics, 10User-Elukey: Add the prometheus jmx exporter to all the Druid daemons - https://phabricator.wikimedia.org/T177459#3659740 (10elukey) [08:41:14] 10Analytics, 10User-Elukey: Add the prometheus jmx exporter to all the Zookeeper daemons - https://phabricator.wikimedia.org/T177460#3659754 (10elukey) [08:42:13] 10Analytics-Cluster, 10Analytics-Kanban, 10monitoring, 10Patch-For-Review, 10User-Elukey: Use Prometheus for Kafka JMX metrics instead of jmxtrans - https://phabricator.wikimedia.org/T175922#3659769 (10elukey) [08:42:15] 10Analytics, 10User-Elukey: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344#3590888 (10elukey) [09:34:12] (03PS2) 10Fdans: Add top articles by pageviews metric [analytics/wikistats2] (develop) - 10https://gerrit.wikimedia.org/r/382139 (https://phabricator.wikimedia.org/T175266) [09:36:04] (03CR) 10Fdans: "@mforns thank you so much for such a great review! I added all the changes you've suggested. Big fan of linking directly to the articles, " (033 comments) [analytics/wikistats2] (develop) - 10https://gerrit.wikimedia.org/r/382139 (https://phabricator.wikimedia.org/T175266) (owner: 10Fdans) [09:36:53] (03CR) 10jerkins-bot: [V: 04-1] Add top articles by pageviews metric [analytics/wikistats2] (develop) - 10https://gerrit.wikimedia.org/r/382139 (https://phabricator.wikimedia.org/T175266) (owner: 10Fdans) [09:43:20] Running errand for a bit + early lunch people, ttl! [10:52:13] Hi a-team - Back in half-half mode - Will spend my day catching up and finishing beginning-of-quarter paperwork [11:12:04] * elukey waves to joal [11:12:13] \o [11:13:28] joal: how are you feeling? [11:14:07] elukey: As if an elephant has been walking over my forehead and belly - kinda [11:14:51] elukey: But it's great, I can do things today, I'm not not anymore in between batroom for me, then for Lino, then caring Naé etc [11:19:29] (03CR) 10Joal: [C: 04-1] "Comments inside. -1 because of path error." (036 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365517 (https://phabricator.wikimedia.org/T170764) (owner: 10Amire80) [11:19:47] elukey: How has it been goind this week? [11:22:02] joal: argh not a nice week for you, so sorry :( glad that you feel better now! [11:22:11] the week has been positive and negative so far [11:22:22] do you want good or bad news first ?:D [11:22:25] hm - not sure what it means [11:22:28] hm, not sure either :) [11:22:33] you decide [11:23:15] so Kafka Jumbo is now configured with basic ACLs and related logging, everything's working fine and we are only missing TLS certificate management to start the game :) [11:23:29] Yay, that seems like a good news :) [11:24:02] I found a way to expose druid metrics to prometheus but it is a bit weird and I'd need to know what you think about it, but not super urgent [11:24:37] Andrew yesterday tried to merge the change to add load balancing to Druid but for various network reasons that he'll explain everything was reverted [11:25:00] and from what I've read there is a fair amount of work to do if we want druid + load balancing [11:25:15] (that was the not-so-good one :P) [11:25:27] mwarf [11:27:47] Thanks for the heads up elukey [11:32:12] I also have some ideas for Hadoop and JVMs, but everything can wait few days [11:33:04] see joal I miss our daily talks in the morning, otherwise I have to keep all my non-sense ideas for me and I feel bored [11:33:15] :) [11:33:53] fdans doesn't like me so he avoid any chat until standup [11:34:21] and Marcel wasn't feeling well this week too [11:34:38] hm, hard week for a-team:( [11:34:40] :P [11:34:40] nooooo lucaaaa [11:34:43] hahaahah [11:34:59] elukey: about druid, LVS not being working also means us not starting to split clusters, right? [11:35:21] especially now, i’ve been doing your hours lately elukey [11:35:27] hellooooo joal [11:36:18] Heya fdans [11:38:23] Always hard when you're not around jo :) [11:38:36] fdans: <3 [11:39:01] Hi milimetric [11:39:04] joal: correct.. I mean we can split the clusters but we need to take some decisions first, Andrew will probably outline all of them during standup [11:55:56] (03PS3) 10Fdans: Add top articles by pageviews metric [analytics/wikistats2] (develop) - 10https://gerrit.wikimedia.org/r/382139 (https://phabricator.wikimedia.org/T175266) [11:56:29] I think this was nuria_ and milimetric 's plan from the beginning [11:57:05] really dislike gerrit => suggest we get a super worse solution => now love gerrit [11:58:57] we've been found out!!! Quick, hide! [11:59:29] :) [12:20:08] question for the druid masters [12:20:23] I am on druid1001 and I am trying to make sense of the logs in /var/log [12:21:54] something like 2017-10-05T00:00:00.000Z should collect all the queries right? [12:22:21] I dont understand that last part elukey [12:23:01] there is a log named "2017-10-05T00:00:00.000Z" [12:23:03] :D [12:23:32] Ah [12:23:34] and from http://druid.io/docs/0.9.2/configuration/index.html I don't see a way to name it differently [12:23:41] can only see druid.request.logging.dir [12:24:56] then one log for each daemon [12:25:04] with content tagged as Event feed [12:26:14] well elukey- looks like you know even more than I do :) [12:26:41] nono I am dumping things in here hoping that somebody will enlight me :D [12:26:59] why am I doing it? I know it looks weird but I need a way to find druid metrics for prometheus :P [12:28:14] elukey: what weould be interesting metrics? [12:30:24] joal: there are the metrics available - http://druid.io/docs/0.9.2/operations/metrics.html [12:34:26] if they are not needed I can only grab the jvm ones from the mbeans via jmx (only ones available grrr) [12:44:06] elukey: I don't think I have seen any of those metrics yet, therefore it's difficult to say [12:44:53] joal: do they look useful from the description? (trying to get an idea if it is worth or not) [12:45:23] elukey: I think I'd like broker-query-time and cahe-oriented data (almost all of them) [12:45:46] elukey: The thing is, I don't know if they are individual per query, aggregated, and how they look like [12:46:00] elukey: Best could be to enable then, and have a look? [12:51:03] I think that they are already enabled, but dumped in log files with text around [12:51:47] druid only emits data, and the "Recipients" can be logfile, http and graphite [12:51:51] Ah, ok [12:51:55] with http it sends a post to and endpoint [12:59:51] 10Analytics, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3660553 (10fdans) a:05hashar>03fdans [13:14:39] fdans: just did git checkout release on thorium for wikistats2, and merged your code review (plus checked the puppet run) [13:15:37] elukey: thank you ma friend :) [13:43:24] fdans: did you end up using release as the "production" branch? [13:43:42] yes, that's the patch that elukey just merged [13:43:49] also updated docs to reflect that [13:43:51] ok, cool [13:44:21] just caught your chat with N too late last night, and wasn't sure [13:44:58] 10Analytics-Kanban, 10Patch-For-Review, 10Release-Engineering-Team (Kanban): Move Wikistats 2 from Differential to Gerrit - https://phabricator.wikimedia.org/T177288#3660665 (10fdans) [13:46:19] elukey: I see druid finished the daily job you restarted. Do you think it's healthy enough to run the monthly or should I still wait? [13:47:14] 10Analytics: Alert user about adblocker preventing AQS requests - https://phabricator.wikimedia.org/T177491#3660684 (10fdans) [13:47:16] milimetric: not a super expert but I'd say yes, the failure looked random [13:47:30] 10Analytics, 10Analytics-Wikistats: Alert user about adblocker preventing AQS requests - https://phabricator.wikimedia.org/T177491#3660698 (10fdans) [13:48:02] k [13:48:39] !log restarted banner_activity-druid-monthly for September again [13:48:41] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:54:37] ottomata: o/ [13:56:33] I am attempting to have a single file containing only json metrics from druid with https://gerrit.wikimedia.org/r/#/c/382452/1/modules/druid/templates/log4j2.xml.erb [13:56:51] that could be completely wrong of course, me and log4j are not close friends [14:18:38] tested the change on druid1006's broker, works like a charm [14:18:59] there is now a file called /var/log/druid/broker-metrics.log with json stuff in there [14:19:23] meanwhile /var/log/druid/broker.log contains only the other generic logging [14:19:34] does it sound reasonable? [14:19:55] o/ analytics folks! [14:20:09] hey [14:20:13] I'm trying to transfer a big file from a university computer onto stat1006 through SCP [14:20:31] It looks like I can't connect to the IP address from the stat machine but I can from my local computer. [14:20:42] The files are too big to pass through my machine. [14:20:45] What do? [14:21:17] elukey: sounds awesome! [14:21:29] \o/ [14:23:11] halfak: hm. [14:23:26] surely the analytics firewall doesn't help [14:23:28] quick idea: can you set up a v simple http server that you could reach on that IP? [14:24:00] ottomata, I was thinking about that but then I'll be fighting with a university to open a port for that. [14:24:01] and then got throught the proxy, good idea indeed [14:24:30] So either I fight with us or fight with an org I have no official affiliation with :/ [14:24:46] You know. What the heck. I'm going to try. Maybe it's already open [14:25:08] It shouldn't be but, shouldn'ts are often ignored in university operations :) [14:26:02] halfak: I haven't played with it but maybe the ProxyCommand ssh config might help [14:27:02] basically doing scp from university to stat1006 but having your local pc as proxy [14:27:23] halfak: aye [14:27:49] yah, but then bytes are still going through his computer, i guess ok cause he's not storing, but still weird [14:27:50] elukey, that could work. I don't like the idea of wasting that bandwidth. The University's VMs are probably really geographically local while I'm not. [14:27:55] But in the end, if that is what's needed. [14:27:55] yeah [14:28:02] I think I'm transfering something like a 1TB [14:28:13] My internet provider is going to destroy me. [14:28:28] elukey: could halfak make another ssh private key from his uni account? maybe just temporary? [14:28:45] then we could allow access from that key, and he could do it the other way around: push from uni server [14:29:11] Oh! I already have that temp key :) [14:29:21] I've been using it just for accessing this one server. [14:32:37] in theory it could be possible to forward the agent to the univ host and use it to authenticate with stat1006, but this wouldn't be super great since anybody with root on univ host will be able to get halfak's ssh agent [14:33:48] let's see if moritzm can help [14:34:07] hi! Sorry to ping you but we'd need a consult for a probably simple scp issue [14:35:47] Maybe I could just connect to a remote ssh host just this once :D [14:36:02] Seems like the easiest solution :) [14:36:06] sure, let me read backlog [14:36:11] No sharing my private key :D [14:36:17] thanks moritzm :) [14:37:14] Reiterating. I need to transfer about 1TB of data from a University machine to stat1006. I'd like to SCP but can't from a stat machine. I've just confirmed that all ports but 22 are blocked by the University in question. [14:38:45] ottomata: anything against me merging https://gerrit.wikimedia.org/r/#/c/382452/2/modules/druid/templates/log4j2.xml.erb and rolling restart druid ? [14:41:15] halfak: not sure I fully understand, are they blocking outgoing, incoming SSH or both? scp $BIGFILE stat1006:~/ fails from you the university host fails? [14:51:59] moritzm, We are blocking outgoing ssh connections [14:52:20] moritzm, I don't want to put my private key on the university's servers. [14:52:27] But I could if you'd rather I do that ;) [14:52:57] It'd be a temp key I'd put in place just for this. [14:54:21] elukey: nope please do! [14:54:38] that is a good idea, i've been doing grep -v to not have to look at those metrics in logs [14:54:48] super [14:57:01] ]' [14:57:07] heh, sorry, elukey wait [14:57:18] maybe let's give this indexing job a chance to finish? [14:57:59] ahahha sure sure [14:58:01] https://hue.wikimedia.org/jobbrowser/jobs/job_1504006918778_137686/single_logs [14:58:21] I wanted to do it after standup [14:58:23] actually, elukey it's just in hadoop now so if you do it fast you can probably do it before the indexing starts [14:58:24] I'll merge in the meantime [14:58:27] nono [14:58:31] not in a hurry [14:58:41] but indexing might take a while [14:58:43] ah, so your concern is that the university machine is a centrally managed host where someone else is root. that's certainly not great, don't you have the possibility to use your notebook in the uni network and run the transfer from there? [15:07:33] 10Analytics-Kanban, 10Operations, 10Patch-For-Review, 10User-Elukey: Tune Kafka logs to register clients connected - https://phabricator.wikimedia.org/T173493#3661052 (10elukey) [15:08:16] 10Analytics-Kanban, 10Analytics-Wikistats: Handle long project names in Wikiselector - https://phabricator.wikimedia.org/T173373#3661057 (10mforns) [15:08:27] 10Analytics-Kanban, 10Patch-For-Review: Replace references to dbstore1002 by db1047 in reportupdater jobs - https://phabricator.wikimedia.org/T176639#3661058 (10mforns) [15:11:06] "notebook in the uni network"? [15:11:09] moritzm, ^ [15:11:17] Oh! like physically be present. [15:11:29] No. not the uni I'm affiliated with. Would be a long flight. [15:11:51] No one there has analytics cluster access. [15:13:01] (03PS1) 10Fdans: Merge branch 'develop' into master [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/382466 (https://phabricator.wikimedia.org/T177288) [15:13:06] one of the big issue that I am seeing now is that stat1006 is in the analytics network and we have special rules on our routers to block traffic going outside the analytics network on port 22 [15:13:15] only few target hosts are allowed [15:13:37] No history of short-term exceptions? [15:13:55] It seems like this would be the most secure strategy for getting the files moved. [15:14:11] we could think about it, I don't see a big issue, I need to figure out though if this would be enough :D [15:14:21] Gotcha. :) [15:14:26] (I mean, if then we'll have other blockers) [15:14:31] so the correct and clean way would be to sync this to a machine you trust (like your notebook) and then sync to stat1006 from that host. however, I can see that this is somewhat impractical if your ISP connection is slow [15:14:35] how urgent it is? [15:15:20] adding an additional temporary key for the transfer from that uni host to stat1006 seems fine to me in terms of risk assessment [15:15:37] but I think this would need to be signed off by Mark [15:15:53] since it's a slight violation of our best practices/guidelines [15:15:54] Hmm... The urgency really comes from how this is blocking me from other work. If I can't get it done this week, it might never happen and that would crash a Research Team goal unless someone else picks it up. [15:16:12] moritzm, agreed. If we could do that, I can get the xfer started in short order. [15:16:44] But yeah, kind of insecure. One option is to delete the key as soon as I start the xfer. That way someone would at least need to dig through memory to find it. [15:16:55] Maybe it's even discarded from memory after the handshake? [15:17:06] Narrow that window :) [15:17:13] DarTar, speak of the devil [15:17:17] :D [15:17:37] Trying to figure out the urgency of transfering meen chul's ref's dataset to stat machines/dataset hosting. [15:17:49] actually, yes you can do that: ssh-agent has the brillantly named "ssh-add -D KEY" command to remove a key from the SSH agent [15:17:52] It's going to take some operations work to make it happen. [15:18:27] hey folks, I gotta head off to the office in a moment, bit hectic here with the girls, I’ll be back later [15:18:32] kk [15:19:08] halfak: I'd say ping Mark on IRC to get his okay and if that's fine Otto or someone from the US opsen should be able to add your temp key during US day time [15:19:46] Oh. Uhh... maybe it'd be better to have a opsen explain it to Mark. I'm gonna be confusing if I explain the plan. [15:22:22] moritzm: super newbie question - say that we'd whitelist the univ external IP addres for ssh on the analytics cr1/cr2 firewall rules, would it be ok for halfak to cp its temp ssh key on stat1006 (the one to log in to the univ), make the transfer and then delete it? Or completely stupid plan due to other reasons? (like no way scp could grab data from outside to production) [15:23:37] this could be done, but wouldn't have much benefit over going through the bastions/scp from what I can tell [15:23:55] that uni should have good outgoing bandwidth I guess [15:24:26] ah so we want to do the other way around from univ to stat1006, got it [15:24:29] thanks :) [15:25:29] elukey: yeah, I think that's ok. could you ping Mark and check with him? I'm afk for a bit now, playground and bringing the kids to bed [15:25:49] thanks to both of you for your help on this :) [15:26:18] sure :) [15:48:18] elukey: that job died anyway - so cluster's all yours [15:48:32] joal said it was an actual problem with data, I'll follow up with him [15:48:45] super [15:49:25] joal: I was looking at this, but it sounds like the errors you mention are logged somewhere else? https://hue.wikimedia.org/oozie/list_oozie_workflow/0061747-170829140538136-oozie-oozi-W/ [15:49:52] 10Analytics, 10ChangeProp, 10EventBus, 10MediaWiki-JobQueue, 10Services (designing): Split ChangeProp metrics by wiki - https://phabricator.wikimedia.org/T175952#3661258 (10fgiunchedi) (apologies about the delay, I completely missed this!) Yeah it is likely statsd isn't going to like a 800x increase. Go... [15:49:52] milimetric: looking for links - will answer soon [15:50:38] halfak: qq, how are those ores schemas? [15:50:50] (not htat I have time to work on it :) ) [15:50:50] milimetric: culprit is that one: https://hue.wikimedia.org/oozie/list_oozie_workflow/0011451-170829140538136-oozie-oozi-W/?coordinator_job_id=0009900-170228165458841-oozie-oozi-C [15:51:02] ottomata, still fighting with awight about them. Nothing formalized in jsonschema yet [15:51:16] Oh wait you are asking about scores. [15:51:17] milimetric: That day has failed (data is available thanks to Spark) [15:51:30] Deployment estimate is Oct 15th. I can get it up on wmflabs now though. [15:51:39] Was planning to do that deployment today ^_^ [15:52:05] (03PS1) 10Milimetric: Add am.wikimedia to the whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/382471 [15:52:41] ah k, no hurry, just was curious because i want to get that in eventstreams [15:52:46] buuut i really don't have time for a while to work on it : [15:52:47] :/ [15:53:05] (03CR) 10Milimetric: [V: 032 C: 032] Add am.wikimedia to the whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/382471 (owner: 10Milimetric) [16:04:36] 10Analytics-Kanban, 10Operations: LVS for Druid - https://phabricator.wikimedia.org/T177511#3661358 (10Ottomata) [16:08:42] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Create Druid public cluster such AQS can query druid public data - https://phabricator.wikimedia.org/T176223#3661390 (10Ottomata) [16:11:23] milimetric: coming? [16:11:26] joal: coming? [16:12:04] yes [16:18:04] 10Analytics: R execution on stat1005 -> 'stack smashing error' - https://phabricator.wikimedia.org/T174946#3661424 (10fdans) p:05Triage>03Low [16:20:11] 10Analytics: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3661446 (10Nuria) [16:20:13] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Make tranquility work with Spark - https://phabricator.wikimedia.org/T168550#3661445 (10Nuria) [16:25:59] 10Analytics: Create small sample mediawiki-history table in MariaDB - https://phabricator.wikimedia.org/T165309#3262603 (10fdans) Let's sqoop out about a month of data from the mediawiki history. Also docs need to be updated to point out that this data is available. [16:26:57] 10Analytics: Create small sample mediawiki-history table in MariaDB - https://phabricator.wikimedia.org/T165309#3661485 (10fdans) [16:27:08] 10Analytics-Kanban: Create small sample mediawiki-history table in MariaDB - https://phabricator.wikimedia.org/T165309#3262603 (10fdans) [16:30:31] 10Analytics: Make Spark 2.1 easily available on new CDH5.10 cluster - https://phabricator.wikimedia.org/T158334#3661510 (10fdans) [16:34:02] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Port Kafka alerts from check_graphite to check_prometheus - https://phabricator.wikimedia.org/T175923#3661520 (10Ottomata) [16:38:21] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: cdh::hadoop::directory (and other hdfs puppet command?) should quickly check if namenode is active before executing - https://phabricator.wikimedia.org/T130832#3661544 (10fdans) [16:43:33] 10Analytics, 10Discovery-Analysis: Get 'sparklyr' working on stats1005 - https://phabricator.wikimedia.org/T139487#2433802 (10fdans) We'll work on this after T158334 is completed [16:45:39] Gone for diner, back after [16:45:53] milimetric: will CR amir's work today [16:45:58] *job, rather [16:46:57] thanks nuria_, appreciated, I know that's likely to slip up in the avalanche of work we have otherwise [16:51:35] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3661618 (10jcrespo) There is now 1TB available, but we have now a memory (swap) problem: https://grafana.wikimedia.org/dashboard/db/server-board?refresh=1m&orgId=1&var-server=dbstore... [16:51:47] halfak, elukey: sounds like it’s solved? [16:53:08] DarTar: techically it should be doable, but to be super strict and follow rules it might be better to open a phab task with all the details and ask Mark/Faidon to approve it [16:53:54] so if you guys could create one we'd just need to wait for the confirmation to proceed [16:54:00] elukey: gotcha, I’ll create one with halfak [16:54:04] DarTar: this is going beyond my allocated time for this work. [16:54:05] Perfect! [16:54:21] I'll keep working so long as I don't need to draft phab tasks. [16:54:23] :D [16:54:54] elukey: is the process above a good description of what should go in the request? [16:55:00] halfak: deal [16:56:13] DarTar: basically what Moritz said earlier on, but I can review it after you have created it [16:56:24] elukey: great [16:57:43] halfak: in the meantime, can you try to test how fast the proxy solution could be? [16:57:54] elukey, proxy is a no go. [16:58:01] My ISP will shut me down [16:58:34] oh ok [16:58:43] They don't really like it when you download and upload 1TB in a few days. :| [16:59:20] I am not really sure why since it will be a regular data transfer, but I trust you :) [17:00:38] elukey: question if you may [17:00:48] nuria_: sure! [17:01:06] elukey: regarding 1002 i do not understand where the memory swap problem comes from after freeing 1 tb [17:01:28] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3661678 (10Marostegui) >>! In T168303#3661618, @jcrespo wrote: > There is now 1TB available, Excellent news! Maybe this ticket can be closed then? }:-) > but we have now a memory... [17:01:58] wow didn't see it [17:02:34] elukey: ah, mnu just updated [17:03:21] not really sure why it is swapping now, but it must be related to the huge alter done [17:04:32] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3661683 (10jcrespo) Don't want to do that without someone from analytics around. Swap usage seems to be going down, probably even faster when it catches up- so I will leave it as is... [17:05:54] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3661684 (10Marostegui) >>! In T168303#3661683, @jcrespo wrote: > Don't want to do that without someone from analytics around. Swap usage seems to be going down, probably even faster... [17:06:56] elukey, halfak: https://phabricator.wikimedia.org/T177521 [17:07:35] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: dbstore1002 /srv filling up - https://phabricator.wikimedia.org/T168303#3661704 (10elukey) Thanks a lot for all the work people, count me in tomorrow if you want to restart it. I agree with the plan, looks good :) [17:14:56] DarTar, halfak - commented in https://phabricator.wikimedia.org/T177521#3661737, let's wait for some feedback [17:16:21] going afk now, but will re-check tomorrow! [17:16:24] nite a-team! [17:16:32] bye elukey! [17:16:46] o/ [17:31:28] thanks elukey [22:26:05] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board: Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#3662739 (10Tbayer) (Moving discussion back here [[https://gerrit.wikimedia.org/r/#/c/379829/ |from Gerrit]]) @mforns has voice... [22:46:18] By chance were there any issues on the analytics cluster on Sept. 15 between 0 and 6 hrs UTC that might have impacted webrequest data moving into Hive? [22:48:08] 10Analytics, 10Patch-For-Review, 10User-bd808, 10cloud-services-team (Kanban): Remove logging from labs for schema https://meta.wikimedia.org/wiki/Schema:CommandInvocation - https://phabricator.wikimedia.org/T166712#3662768 (10Nuria) @bd808: sounds good, let us know when calls are no loger coming in and we... [22:59:48] (03CR) 10Nuria: "Added some comments, have we tested this job runs?" (034 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/365517 (https://phabricator.wikimedia.org/T170764) (owner: 10Amire80) [23:11:08] AndyRussG: I was looking at your FR ticket but your last comment does not quite add up, there is no data loss on our end (we monitor for that) [23:13:00] AndyRussG: what do you see in September 15th? there are some varnish restarts that I see on SAL but that's just about [23:14:21] AndyRussG: pageviews look stable: http://bit.ly/2z1bh9W [23:30:38] hey nuria_ Mmm which ticket? There are a few... [23:33:32] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Understand differences in results between "Top Authors" on Overview page vs. "Submitters" on Gerrit page - https://phabricator.wikimedia.org/T177566#3663034 (10Aklapper) [23:36:12] 10Analytics-Tech-community-metrics, 10Developer-Relations (Oct-Dec 2017): Understand differences in results between "Top Authors" on Overview page vs. "Submitters" on Gerrit page - https://phabricator.wikimedia.org/T177566#3663047 (10Aklapper) 05Open>03Resolved p:05Triage>03High Ah. Okay. I went to ht... [23:37:37] nuria_: the main issue is that donations rates seem OK, but impression rates (as queried via Druid) are all fluctuating on a daily basis and on average down [23:37:49] Was just going to query directly via Hive [23:37:53] instead of Druid [23:38:32] We're by no means convinced it's an issue with an analytics pipeline, just thought I'd check to see if there anything obvious jumped out :)