[00:17:43] (03PS1) 10GoranSMilovanovic: Semantics Dashboard + File Structure [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/385929 [00:17:44] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Semantics Dashboard + File Structure [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/385929 (owner: 10GoranSMilovanovic) [00:24:36] (03PS1) 10GoranSMilovanovic: Minor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/385930 [00:25:21] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] Minor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/385930 (owner: 10GoranSMilovanovic) [00:26:01] (03PS1) 10GoranSMilovanovic: minor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/385931 [00:26:29] (03CR) 10GoranSMilovanovic: [V: 032 C: 032] minor [analytics/wmde/WDCM] - 10https://gerrit.wikimedia.org/r/385931 (owner: 10GoranSMilovanovic) [01:57:20] 10Analytics, 10Analytics-Wikistats, 10Wikimedia-Site-requests: Add li: Wikibooks to stats.wikimedia.org - https://phabricator.wikimedia.org/T165634#3702458 (10Dcljr) [01:58:08] 10Analytics, 10Analytics-Wikistats, 10Wikimedia-Site-requests: Add li: Wikibooks to stats.wikimedia.org - https://phabricator.wikimedia.org/T165634#3272286 (10Dcljr) Have taken the liberty of renaming the task to clarify what's being requested. [09:58:03] (03PS1) 10Fdans: Only queries breakdowns when they are to be visualised [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/385954 (https://phabricator.wikimedia.org/T178461) [10:04:02] 10Analytics, 10Analytics-Wikistats: Handle negative values in charts - https://phabricator.wikimedia.org/T178797#3702916 (10fdans) [11:14:08] this is so quiet today [11:14:16] shhhhhh ;) [11:14:44] HELLOOOOO JOSEPH [11:14:56] * joal open an eye [11:15:04] HELLOOOOOOOOOO ! [11:15:06] :D [11:16:02] * elukey lunhc! [11:16:06] *lunch [11:16:09] hello people :) [11:16:13] Hi elukey :) [11:23:13] 10Analytics-Cluster, 10Analytics-Kanban, 10Language-Team, 10MediaWiki-extensions-UniversalLanguageSelector, and 3 others: Migrate table creation query to oozie for interlanguage links - https://phabricator.wikimedia.org/T170764#3703003 (10Amire80) Thanks a lot for the updates. I'm traveling at the moment,... [12:40:09] elukey: heya - Would today be the time we deploy new AQS? [12:50:19] sure [12:57:01] joal: whenever you prefer [13:08:17] elukey: is now a good time? [13:11:17] sure [13:16:44] ok, moving forward then [13:17:43] elukey: How do we proceed ? You update the conf manually on aqs1004, I deploy canaray, then we do one by one? [13:19:21] joal: I can merge https://gerrit.wikimedia.org/r/#/c/385339/, enable puppet only on aqs1004, let you deploy/check, then apply it to all the other nodes before you complete the deployment [13:19:33] elukey: works for me :) [13:20:04] joal: all right gimme 2 mins [13:20:14] sure elukey [13:28:00] morning [13:28:31] Hi milimetric [13:29:05] joal: aqs1004 ready to go (the others all have puppet disabled) [13:29:13] elukey: Okey ! [13:30:35] !log deploy AQS from tin [13:30:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:31:28] afaics only 200/404 on aqs1004 now [13:34:25] joal: AQS alarm in ops [13:34:42] yup elukey - deploy failed [13:35:11] elukey: error with druid conf I think [13:35:18] 13:30:46 [aqs1004.eqiad.wmnet] Check 'endpoints' failed: /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-id}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) is CRITICAL: Test Get daily edits for english wikipedia page 0 returned the unexpected status 404 (expecting: 200) [13:35:30] elukey: --^ [13:35:35] elukey: rollbacking? [13:36:36] joal: everything works except for the new druid check right? [13:36:58] elukey: I don't know if other checks work [13:37:02] give me aminute [13:37:14] I am going to depool aqs1004 just in case [13:38:03] ok done [13:39:22] so joal we can try to figure out what's wrong and not rollback puppet for the moment [13:39:46] ok elukey [13:40:00] o/ [13:40:00] elukey: I thought scap depooled machines while deploying? [13:40:06] hi ottomata [13:40:48] joal: yep but after you rollback then it is added back and the config.yaml is wrong :) [13:41:03] elukey: I have not rollbacked yet [13:41:21] elukey: I was trying to understand what's wrong [13:41:34] joal: sure, so I have explicitly depooled it [13:41:41] :) [13:43:45] elukey: warnings in logstash :( [13:45:37] can you paste the link? [13:45:50] elukey: https://logstash.wikimedia.org/app/kibana#/dashboard/default?_g=h@44136fa&_a=h@a4aff35 [13:46:51] doesn't load.. is it a dashboard or discovery? [13:47:49] elukey: I search for AQS in the default dashboard [13:48:45] ah you mean ServiceRunner.run() is deprecated, and will be removed in v3.x etc.. [13:49:02] also Schema hash mismatch: [13:49:03] elukey: no, schema hash warning [13:49:13] loely [13:49:15] *lovely [13:49:20] indeed [13:50:16] elukey: But this doesn't really help - Why the heck a 404 :( [13:50:28] 10Analytics-Kanban: Add Tilman to analytics-admins - https://phabricator.wikimedia.org/T178802#3703198 (10Ottomata) [13:50:41] 10Analytics-Kanban, 10Operations, 10Ops-Access-Requests: Add Tilman to analytics-admins - https://phabricator.wikimedia.org/T178802#3703216 (10Ottomata) [13:52:19] host: druid1004.eqiad.wmnet should be replaced with the lvs endpoint after this, writing down notes [13:52:26] yup [13:53:02] elukey: need to run for unexpected baby need - Will continue to investigate when back [13:53:08] okok [13:53:11] elukey: sorry for that :( [13:53:28] nah we'll sort this mess out, it happens :) [13:59:40] helloooo [14:09:49] 10Analytics, 10Analytics-Dashiki: Add option to not truncate Y-axis - https://phabricator.wikimedia.org/T178602#3697395 (10Milimetric) ooh, that's not good. I'll make a patch, thanks for the report. [14:11:33] fdans: can I catch up with you on wikistats? [14:12:08] milimetric: hell yea, cave? [14:13:06] 10Analytics, 10Research: geowiki data for Global Innovation Index - 2017 - https://phabricator.wikimedia.org/T178183#3703305 (10Milimetric) @leila: no changes have happened at all. However, we are working on migrating the logic to Hadoop this and next quarter. We'll keep this in mind so it makes the GII work... [14:13:12] fdans: cave [14:20:35] I'm back elukey [14:21:04] elukey: Do you think we could update aqs conf on aqs1004 to log info instead of warn to help debugging? [14:25:34] joal: it is a bit hacky but we can try [14:25:53] elukey: it means updating /etc/aqs/config.yaml and restarting aqs [14:26:21] elukey: just making sure I actually know what it means as well :0 [14:26:24] yes yes [14:26:39] so have you rolled back the code or is it still the new one? [14:26:57] (just to understand [14:27:13] it's still the new one [14:27:28] I'd like to try to unde3rstand why the druid backend doesn't work with the new config [14:27:50] elukey: that druid part is the only thing we could not test on deployment - obviously it breaks [14:28:37] joal: aqs1004 ready for your test [14:35:06] elukey: I can't find anything using logstash :( [14:35:27] joal: did you re-run the tests that lead to 400s? [14:35:43] elukey: I ran a query, yes [14:36:12] and did you get a 404 instead of a 200 ? [14:36:19] Yes [14:36:52] mmmmm [14:36:59] did we see that query on druid1004? [14:37:14] I have not checked [14:38:49] I am trying to figure out if Druid returns a 404 because of some weird reason or if it is aqs [14:39:14] elukey: In druid logs, I see requests [14:40:25] so since success:true it might be aqs doing weird things [14:41:14] is the test ok? [14:41:19] elukey: weird thing though: queries I see on druid1004 seems to only be test ones, not the ones I manually do [14:41:28] ah! [14:41:55] That's even weirder ! [14:43:00] joal: can you retry to do some queries towards aqs1004? [14:43:05] I am using tcpdump [14:43:19] in theory I should see traffic to 10.64.0.35 [14:43:24] (druid1004) [14:43:25] (03CR) 10Milimetric: "cool, just a naming/comment comment" (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/385186 (https://phabricator.wikimedia.org/T178461) (owner: 10Fdans) [14:44:43] elukey: starting now [14:44:59] elukey: I tried 3 requests [14:45:09] didn't see anything [14:45:10] mersi milimetric!! [14:45:12] hm [14:45:48] haha [14:46:18] hmm. curious things happening on stat1005 [14:47:12] elukey: will try to repro locally [14:50:11] dsaez: o/ - I had to kill your python process on stat1005 :( [14:50:24] it was eating a ton of memory [14:51:32] :( [14:51:43] I need to move that proccess to spark [14:53:04] 10Analytics-Kanban, 10Operations, 10Traffic, 10Patch-For-Review: Invalid "wikimedia" family in unique devices data due to misplaced WMF-Last-Access-Global cookie - https://phabricator.wikimedia.org/T174640#3703376 (10BBlack) 05Open>03Resolved I'm assuming there's nothing left to do here, re-open otherw... [14:54:08] elukey: just to know, which is the criteria to kill process? there was a Java process eating all the CPU during the whole weekend (and also past weekend), stat1005 was almost not working [14:55:32] dsaez: in #operations stat1005 alarmed and your process was the (as far as I saw) the biggest consumer of memory [14:55:54] after killing it it freed half of the host ram [14:56:13] the OOM killer would have done the job in a bit almost for sure [14:57:04] ok, so using all CPU is ok, but not memory? [14:57:05] dsaez: one of the things that we'll work on probably is resource consume limits for the stat boxes, so the users will know exactly their boundaries and no surprise will happen [14:57:42] dsaez: it is definitely not ok but it doesn't cause any sort of icinga alerts to fail due to the absence of memory [14:57:52] 10Analytics, 10Research: geowiki data for Global Innovation Index - 2017 - https://phabricator.wikimedia.org/T178183#3703407 (10leila) [14:58:11] 10Analytics, 10Research: geowiki data for Global Innovation Index - 2017 - https://phabricator.wikimedia.org/T178183#3683651 (10leila) Thanks, @Milimetric . [14:58:20] there are a lot of cores on those boxes, but I get your point of not being usable [15:00:15] ok, I see. During the weekend doing 'ls' in my home folder was taking around 1 min, due the CPU usage, I was moving all process to stat1006, but given that there is not access to the dumps, plus no sshfs, I needed to go back to stat1005 [15:00:21] a-team: standup? [15:00:37] coming! [15:01:52] 10Analytics, 10Operations, 10Traffic: Artificial spike in offset of unique devices from November to February 6th on wikidata - https://phabricator.wikimedia.org/T165560#3270089 (10BBlack) Is this something we still need answers for, or have we just moved past it into a new normal? [15:07:56] 10Analytics, 10Operations, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#3703436 (10BBlack) We abandoned the original intent of this ticket, I think? [15:16:59] 10Analytics-Kanban, 10Operations, 10Ops-Access-Requests: Add Tilman to analytics-admins - https://phabricator.wikimedia.org/T178802#3703491 (10Nuria) Approved on my end. [15:32:55] dsaez: when you see those slowdowns feel free to drop me an email, I'll try to double check [15:33:14] an ls taking one min is definitely not ok [15:33:45] 10Analytics, 10Operations, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#3703590 (10Nuria) Ticket can be closed. [15:36:55] 10Analytics, 10Analytics-Wikistats: Handle negative values in charts - https://phabricator.wikimedia.org/T178797#3703596 (10fdans) [15:37:52] 10Analytics-Kanban, 10Analytics-Wikistats: Handle negative values in charts - https://phabricator.wikimedia.org/T178797#3702916 (10fdans) [15:39:14] 10Analytics-Dashiki, 10Analytics-Kanban: Add option to not truncate Y-axis - https://phabricator.wikimedia.org/T178602#3703601 (10fdans) a:03Nuria [15:39:34] 10Analytics, 10Analytics-Wikistats: Wikistats: Corners of dashboard miniatures overflow when no data - https://phabricator.wikimedia.org/T178812#3703603 (10mforns) [15:40:02] 10Analytics-Dashiki, 10Analytics-Kanban: Add option to not truncate Y-axis - https://phabricator.wikimedia.org/T178602#3697395 (10fdans) [15:40:57] 10Analytics, 10Analytics-Dashiki, 10Analytics-Kanban: Add option to not truncate Y-axis - https://phabricator.wikimedia.org/T178602#3697395 (10fdans) [15:41:25] 10Analytics-Dashiki, 10Analytics-Kanban: Add option to not truncate Y-axis - https://phabricator.wikimedia.org/T178602#3703630 (10fdans) [15:42:55] 10Analytics, 10Analytics-Wikistats: Beta release: Wikistats: Corners of dashboard miniatures overflow when no data - https://phabricator.wikimedia.org/T178812#3703640 (10fdans) [15:50:33] 10Analytics, 10Operations, 10Traffic: A/B Testing solid framework - https://phabricator.wikimedia.org/T135762#3703672 (10BBlack) 05Open>03Resolved a:03BBlack [15:50:45] elukey: ok, I'll do, thanks [15:54:45] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Datasets-Webstatscollector, 10Language-Team, and 5 others: Investigate anomalous views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#3703684 (10fdans) [15:56:58] 10Analytics, 10Analytics-Kanban: Beta Release: Support Annotations on Wikistats 2.0 graphs - https://phabricator.wikimedia.org/T178813#3703686 (10Nuria) [15:59:41] 10Analytics, 10Easy: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604#3703708 (10fdans) [16:05:35] 10Analytics, 10MediaWiki-API, 10RESTBase-API, 10Services (blocked): Top API user agents stats - https://phabricator.wikimedia.org/T142139#3703727 (10fdans) a:03Nuria [16:07:45] elukey: I have a patch - it's actually in config [16:08:10] joal: ready when you are [16:08:58] elukey: I think we should couple the patch with LVS usage: basically hyperswitch disallow external HTTP code by default, and you need to explicitely configure it if you want ot have some [16:09:32] elukey: to be good, we'd set the LVS adress as the only external accepted URI [16:10:08] We can go for druid1004 for now, but since we're updating conf, maybe going for LVS would be better? Or I can also hear that less moving pieces is better [16:10:11] You tell me [16:10:14] elukey: --^ [16:10:27] I am fine with it [16:10:32] (LVS config) [16:10:35] ok [16:11:28] joal: what does it mean "external" HTTP btw for hyperswitch? I mean, for me the LVS endpoint or druid1004 is internal [16:11:42] but probably my internal definition is not the same :D [16:12:05] correct elukey :) [16:12:15] it's all about the definition of internal :) [16:12:31] So, hyperswitch is like an endpoint-provider [16:12:58] endpoints defined inside hyperswitch are "internal" to hyperswitch itself [16:13:15] 10Analytics, 10EventBus, 10Wikimedia-Stream, 10MW-1.30-release-notes (WMF-deploy-2017-07-25_(1.30.0-wmf.11)), and 2 others: Add parsedcomment to recentchange stream - https://phabricator.wikimedia.org/T170145#3703736 (10fdans) 05Open>03Resolved a:03fdans [16:13:17] which means http calls to prod-internal endpoints are external to hyperswitch [16:13:36] makes sense elukey ? [16:14:18] joal: thanks a lot, it does now :) [16:14:34] so +1 for your idea about the druid lvs "external" endpoint :D [16:14:55] elukey: with that explanation, it also makes sense that tests don't fail: they are using a fake druid built inside hyperswitch (therefore internal !!!0 [16:14:58] How bad [16:15:10] !11!!!11!!! [16:15:48] 10Analytics, 10EventBus, 10Wikimedia-Stream: Add tags to recentchange stream - https://phabricator.wikimedia.org/T171182#3456794 (10fdans) @Nirmos can you elaborate on exactly what tags are you expecting? [16:15:55] elukey: do I try to patch AQS config, or do I let you do it? [16:16:50] joal: if you have time I'd ask you to do it, you have more context than me [16:16:58] elukey: trying so :) [16:17:27] elukey: for LVS, it's only a horizon change, right? [16:19:14] 10Analytics-Cluster, 10Analytics-Kanban: Filter local IPs before checking for geo info - https://phabricator.wikimedia.org/T160822#3703756 (10fdans) [16:19:25] 10Analytics, 10Research: geowiki data for Global Innovation Index - 2017 - https://phabricator.wikimedia.org/T178183#3703758 (10Rafaesrey) Thank you, Leila, for all these updates. Best, Rafael. [16:19:40] joal: do you mean for labs? [16:20:32] anyhow, it should be only a hiera change [16:20:54] for the aqs.yaml file [16:21:46] Got it [16:27:36] elukey: In puppet, do we have an easy way to escape charaters (precisely, slashes) to be usable in regex pattern? [16:28:59] 10Analytics: Productionize analysis of editcount vs per_user_revision_count - https://phabricator.wikimedia.org/T168648#3703791 (10Milimetric) [16:32:41] joal: good question, not sure.. what do you need to eascape? [16:32:44] *escape [16:32:47] 10Analytics, 10EventBus, 10Wikimedia-Stream: Add tags to recentchange stream - https://phabricator.wikimedia.org/T171182#3703814 (10Nirmos) ChangeTags as described on https://www.mediawiki.org/wiki/Manual:Tags. They are tags that are attached to edits. They can either come from MediaWiki (like `visualeditor`... [16:33:03] elukey: I'm building a url pattern from host info [16:33:16] elukey: might be easier to actually add a variable [16:33:21] elukey: thoughts? [16:34:24] I'd need some examples :) [16:34:37] elukey: batcave for a minute: [16:34:39] ? [16:35:57] ops meeting :( [16:35:58] 10Analytics: vet edit data on the data lake - https://phabricator.wikimedia.org/T153923#3703825 (10Erik_Zachte) I collected feedback in https://phabricator.wikimedia.org/T178591 (I don't know how to link it here as a subtask, I never did such) [16:40:13] elukey: the pattern in AQS config to accept connections will be: ^http\/\/druid-public-broker.svc.eqiad.wmnet:8082\/druid\/v2\/$ [16:41:41] Which means, if I want to build it from existing values, I need to escape '/' to '\/' in druid_query_path [16:42:14] For the moment I have added a variable, but it's not super nice since it's really just an escape [16:43:36] joal: let's create the code review and see what the pcc compiler says [16:48:08] elukey: patch on his way [16:48:25] elukey: I finnaly didn't add LVS, to better separate concerns [16:48:43] I'll provide a new patch with LVS update when this first one is ready :) [16:48:55] surez [16:48:57] elukey: I let you read, go for diner with Lino for 1/2 hour and then back [16:51:06] sure [16:57:52] 10Analytics-Kanban, 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Add Tilman to analytics-admins - https://phabricator.wikimedia.org/T178802#3703882 (10Ottomata) [17:01:50] 10Analytics-Kanban, 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Add Tilman to analytics-admins - https://phabricator.wikimedia.org/T178802#3703198 (10RobH) Please note adding Tilman to analytics-admins access was approved in today's operations meeting. [17:01:59] 10Analytics, 10Analytics-Wikistats: Feedback on hive table mediawiki_history by Erik Z - https://phabricator.wikimedia.org/T178591#3703911 (10Erik_Zachte) {F10386554} Another comparison between Wikistats 1 and 2: this time edit counts per user. I compared edit counts for users with 5000+ edits on af.wikipedi... [17:17:43] bearloga: of interest, explanation of the whys of data lake: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Data_Lake/Edits/Pipeline/Page_and_user_history_reconstruction [17:18:45] 10Analytics, 10EventBus, 10Wikimedia-Stream: Add tags to recentchange stream - https://phabricator.wikimedia.org/T171182#3703952 (10Krinkle) [17:19:52] 10Analytics, 10EventBus, 10Wikimedia-Stream: Include MediaWiki change tags in RecentChange event stream - https://phabricator.wikimedia.org/T171182#3456794 (10Krinkle) [17:20:03] nuria_: very cool, thank you! [17:26:37] 10Analytics, 10EventBus, 10Wikimedia-Stream: Include MediaWiki change tags in RecentChange event stream - https://phabricator.wikimedia.org/T171182#3456794 (10Krinkle) @Nirmos The EventStreams system, just like the RCStream system previously, is a public service for storing, buffering and distributing the RC... [17:27:10] 10Analytics, 10EventBus, 10Wikimedia-Stream: Include MediaWiki change tags in RecentChange event stream - https://phabricator.wikimedia.org/T171182#3703974 (10Krinkle) [17:27:29] 10Analytics, 10EventBus, 10Wikimedia-Stream: Include MediaWiki change tags in RecentChange event stream - https://phabricator.wikimedia.org/T171182#3456794 (10Krinkle) [17:27:39] joal: just submitted the last version of the aqs patch [17:27:50] if you want to double check it we can then deploy [17:30:42] elukey: problem is that you changed the pattern for LVS, but not the host !! [17:31:59] good point! [17:32:02] elukey: tomorrow my morning, would you have some time to help with prometheus mirror maker stuff? [17:32:41] nuria_: ping :) [17:32:45] ottomata: of course! [17:32:47] gr8 :) [17:32:59] joal: fixed! [17:33:47] joal: I also removed the double // ending chars in the pattern, hope that it wasn't intented, otherwise I'll fix it [17:45:21] elukey: it actually was important, it's perl regexp style :) [17:47:37] joal: sure but two in the end? pattern: /<%= @druid_uri_pattern %>// [17:47:52] I am probably super ignorant, if this is the case I'll re-add it [17:47:59] Yes sir :) /pattern/replacement/ [17:48:09] if not a replacement, /pattern// [17:48:09] :) [17:50:13] see less ignorant now :) [17:50:18] fixing [17:54:17] joal: https://puppet-compiler.wmflabs.org/compiler02/8425/aqs1004.eqiad.wmnet/ [17:54:30] pattern is now - pattern: /^http\/\/druid-public-broker.svc.eqiad.wmnet:8082\/druid\/v2\/$// [17:58:03] awesome elukey :) [17:58:39] joal: if you are ok with the changes joal I can merge and deploy [17:59:07] elukey: let's go :) [17:59:14] elukey: on aqs1004 only please ;) [17:59:19] with manual restarst? [17:59:39] yeppa [18:04:32] joal: we are good on aqs1004 [18:06:18] elukey: error is different, but still no data [18:06:35] elukey: I assume it's a LVS thing, but can't be sure [18:07:45] what is the error? [18:08:24] no data :( [18:09:04] elukey: so it's like we can request, but we get no data as result (which is why I suspect LVS) [18:09:34] I tried elukey@aqs1004:~$ curl http://druid-public-broker.svc.eqiad.wmnet:8082/druid/v2 -X POST (that is of course wrong) and I get a error in return [18:09:37] from druid [18:09:45] so it shouldn't be the LVS [18:10:06] joal: I can hack a second config.yaml and test with druid1004 as host [18:10:15] to rule out lvs [18:10:18] elukey: you're my knig [18:10:21] king [18:10:29] * joal loves knig :D [18:11:22] joal: done! [18:11:24] :D :D :D [18:12:12] elukey: arf, fail !!!! [18:12:38] same error? no data? [18:14:28] correct elukey u [18:14:42] joal: can you retry (or paste in here) the query that you are trying? I'd like to see from tcpdump if this time data flows [18:15:11] elukey: I do curl http://localhost:7232/analytics.wikimedia.org/v1/edits/aggregate/de.wikipedia/all-editor-types/all-page-types/monthly/20160101/20160401 [18:15:56] no data flowing :( [18:16:03] Meeeeeh [18:16:44] elukey: the pattern I used home that, with tunnel, worked: /^https?:\/\/localhost:8082\/druid\/v2// [18:18:09] current pattern is /^http\/\/druid1004.eqiad.wmnet:8082\/druid\/v2\/$// [18:18:17] so no $ in the end [18:18:24] elukey: I'd assume that yes [18:18:42] ahhh wait maybe it is the $ ? [18:19:05] elukey: I'd say so :S [18:19:52] tried now, seems not working pfff [18:20:19] elukey: last \/ ?? [18:20:23] bearloga, chelsyx: i looked at data from wikistats regarding bots/users in http://stats.wikimedia.org for commons and I think it will be worth double checking with data you are working on [18:21:43] joal: nope.. [18:21:48] bearloga, chelsyx commented on wiki but i think looking at monthly series (for 2017/2016) is probably of value [18:22:00] nuria_: Thanks! I saw your response for T177354 and I'm writing a comments now [18:22:00] T177354: Metrics for SDoC: look at contributions - https://phabricator.wikimedia.org/T177354 [18:22:02] elukey: I have it: COLONS !!! [18:22:18] elukey: http --> : <-- // [18:22:50] ottomata, do you have some time to brainbounce on EventLoggingToDruid? cc joal :] [18:24:37] yes for sure [18:24:38] gimme 2 mins [18:25:26] joal: nope :( [18:25:29] chelsyx: ok, good, will look arround some more on thsoe tickets see if anything rings a bell of data we might alredy have [18:25:34] elukey: REALLY ??? [18:25:42] *already [18:26:06] nuria_: Thanks! :) [18:26:19] joal: so this one doesn't work pattern: /^http:\/\/druid-public-broker.svc.eqiad.wmnet:8082\/druid\/v2\/$// [18:26:30] elukey: k [18:26:48] $ at the end is probably not correct [18:28:12] elukey: I don't understand man :( [18:30:16] so mobileapps is broken and I hope it is not aqs otherwise it will be a big WTF :D [18:30:22] removing also $ [18:32:53] joal: pattern: /^http:\/\/druid-public-broker.svc.eqiad.wmnet:8082\/druid\/v2\/// doesn't work [18:36:13] elukey: :( [18:37:34] elukey: if you need to stop for tonight I'll rollback and we'll get back to it tomorrow [18:38:19] elukey@aqs1004:~$ curl "http://localhost:7232/analytics.wikimedia.org/v1/edits/aggregate/de.wikipedia/all-editor-types/all-page-types/monthly/20160101/20160401" [18:38:20] mforns: sorry! [18:38:21] ya let's bc [18:38:22] {"items":[{"project":"de.wikipedia","editor-type":"all-editor-types","page-type":"all-page-types","granularity":"monthly","results":[{"timestamp":"2016-01-01T00:00:00.000Z","edits":1229936},{"timestamp":"2016-02-01T00:00:00.000Z","edits":1098262},{"timestamp":"2016-03-01T00:00:00.000Z","edits":973694}]}]} [18:38:22] ready? [18:38:27] joal: yesssssss [18:38:32] it was the trailing / !!! [18:38:40] ottomata, yes! [18:38:52] elukey: Maaaaaan [18:38:56] elukey: I'm so sorry :( [18:39:20] elukey: I thank you a lot for finding the matching pattern :) [18:39:55] nuria_: the wks2 vetting meeting falls during monthly metrics :( [18:39:59] nuria_: can we move it? [18:44:05] joal: https://gerrit.wikimedia.org/r/#/c/386038/1/hieradata/role/common/aqs.yaml - +1 ? [18:44:49] elukey: Yes ! [18:47:05] milimetric, nuria_: This is so WKS2-vetting: https://xkcd.com/1906/ [18:47:54] hahaha [18:47:58] ok joal all done, merged and run puppet on aqs1004 [18:48:05] shall we deploy again and see? [18:48:14] elukey: sure [18:48:23] elukey: That means, rollback then deploy again, right? [18:49:00] joal: it could be an option yeah [18:49:14] cause now I have a failing deploy still running [18:50:08] yes yes it makes sense [18:50:13] ok, doing elukey [18:50:40] !log Deploying AQS after fix [18:50:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:52:08] elukey: actually, rollback will fail as well :( [18:54:04] it seems working fine now! [18:54:13] it does :) [18:54:17] Thanks again a lot elukey :) [18:54:40] all right before saying "Victory" let's also deploy on the rest of the nodes :D [18:54:49] currently doing so elukey [18:54:53] nooo [18:54:58] need puppet run first :) [18:55:01] Arffffff :( [18:55:12] aqs1005 currently doing [18:55:18] and I'll wait for others [18:55:18] lemme run puppet [18:55:20] dsure [18:56:51] all set [18:56:57] elukey: all nodes? [18:57:08] yepp [18:57:29] ok - dpeloyemnt failed for aqs1005 - Is there a way to redo it easily? [18:57:44] may no rollback, then scap deploy --limiaqs1005? [18:57:50] yeah [18:57:53] going to depool it [18:57:55] k [18:58:07] done [18:58:16] actually deploy failed anyhow [18:58:32] so relaunched it [18:58:34] :( [18:59:03] elukey: no need to deplool [18:59:11] good [19:01:02] elukey: Brand new AQS everywhere : [19:01:04] :) [19:01:06] \o/ [19:01:09] * elukey dances [19:01:13] Man, that's been not nice [19:01:30] * joal bow in silence to elukey's mantra :) [19:01:56] elukey: we still have warnings, but they don't prevent us to run [19:02:04] I'll ask services tomorrow how to update [19:02:58] yeah let's open a task and see tomorrow [19:03:03] everything looks good afaics [19:03:08] Agreed [19:03:11] (httpry on the host and metrics) [19:03:34] and we are using the lvs endpoint!! [19:03:35] finally [19:04:11] yes sir :) [19:04:23] all right if all is done I'd log off for today :) [19:04:25] we are getting closer to expected stuff ! [19:04:35] sure elukey, see you tomorrow :) [19:04:41] byyyeee o/ [19:04:43] * elukey off [19:05:09] * elukey cheers for team aqs/cassandra winning again :) [19:05:15] :D [19:06:59] 10Analytics-Kanban: Investigate AQS cassandra schema hash warninga - https://phabricator.wikimedia.org/T178832#3704349 (10JAllemandou) [19:09:31] ok team, after that difficult aqs taming, I'm gone as well [19:09:34] see you tomorrow :) [19:13:50] 10Analytics, 10Analytics-Wikistats: Feedback on hive table mediawiki_history by Erik Z - https://phabricator.wikimedia.org/T178591#3704371 (10leila) [19:13:52] 10Analytics: vet edit data on the data lake - https://phabricator.wikimedia.org/T153923#3704370 (10leila) [19:16:30] 10Analytics: vet edit data on the data lake - https://phabricator.wikimedia.org/T153923#3704372 (10leila) @Erik_Zachte I just added it as a subtask. For the future, you can click on Edit Related Task on the top right-hand-side of your page and then click on Edit Subtasks and add the phabricator ticket number to... [20:10:03] joal: moved now [20:27:23] 10Analytics-Kanban, 10Operations, 10Ops-Access-Requests, 10Patch-For-Review: Add Tilman to analytics-admins - https://phabricator.wikimedia.org/T178802#3704705 (10Tbayer) 05Open>03Resolved Thanks all! [21:37:53] (03CR) 10Milimetric: [C: 04-1] Only queries breakdowns when they are to be visualised (038 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/385954 (https://phabricator.wikimedia.org/T178461) (owner: 10Fdans) [22:11:39] 10Quarry, 10Cloud-Services: Error: BIGINT UNSIGNED value is out of range - https://phabricator.wikimedia.org/T178848#3704967 (10Huji)