[00:03:35] Quarry, AutoWikiBrowser, WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2255546 (IKhitron) [08:16:57] * elukey self review mode (sigh), ping me if needed! [09:06:27] Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2256089 (jcrespo) This is done for me, waiting for confirmation to resolve the ticket: ``` MariaDB dbstore1002.eqiad.wmnet log > SELECT event_userEditCount, editCountBucket FROM MobileWebCta_597... [09:06:47] Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2256090 (jcrespo) [09:14:06] Analytics, DBA: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2256094 (jcrespo) > we still need to decide what and how If it is very dynamic, I would recommend setting a table with table names, that way an event could read it and actuate depending on that. Alternat... [10:09:30] hi team [10:09:31] :] [10:13:04] mforns: o/ [10:13:10] if you want we can deploy EL [10:13:11] hey elukey [10:13:16] hehe ok! [10:18:31] Analytics, DBA: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#2256233 (mforns) @jcrespo I think you're right, the latter seems to have better visibility and also permits to CR changes, which will be good, given that we'll be changing security settings. Let me know... [10:21:58] a-team: as gentle reminder, I am going to start the work on stat1001 at 14:00 CEST (~1.5 hrs from now) [10:37:45] elukey, BTW, to activate the banner in all Dashiki instances, the only thing needed is editing this page: https://meta.wikimedia.org/wiki/Dashiki:OutOfService and replacinf "false" with "true". And when done, replace it back to "false". [10:38:26] elukey, if I'm around, just ping me and I'll do that [10:49:20] mforns: WOA, thanks! [10:49:57] elukey, I'm trying to deploy EL but there's an error, I don't think it's critical, but prevents me to deploy EL [10:50:00] np :] [10:50:44] mforns: which one? [10:50:51] we can batcave if you want [10:51:00] elukey, ok [10:51:05] omw! [11:21:07] !log deployed last version of Event Logging. Service also restarted. [11:21:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [11:27:07] mforns: I have been seeing "(Additional properties are not allowed (u'varnish4hits', u'varnish4' were unexpected))" while tailing in the past days EL and I was wondering if we are discarding valid data [11:27:59] * mforns looks [11:28:42] thanks! [11:28:48] going to grab something for lunch! [11:32:41] buon appetito elukey [11:49:31] Analytics-EventLogging, Analytics-Kanban, Patch-For-Review: EventLogging dies when fetching a schema over HTTP that does not exist. {oryx} - https://phabricator.wikimedia.org/T124799#2256347 (mforns) [12:05:42] !log enabled maintenance banner to dashiki based dashboards via https://meta.wikimedia.org/wiki/Dashiki:OutOfService [12:05:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [12:14:56] !log deployed Varnish change to force HTTP 503 for datasets.wikimedia.org, stats.wikimedia.org, metrics.wikimedia.org as prep-step for OS reimage. [12:14:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [12:23:38] what the hell, I don't see varnish changes running puppet [12:23:39] grrr [12:56:44] Analytics-EventLogging, MediaWiki-extensions-MultimediaViewer: MultimediaViewerNetworkPerformance schema validation errors - https://phabricator.wikimedia.org/T134165#2256431 (Gilles) [12:57:24] Analytics-EventLogging, MediaWiki-extensions-MultimediaViewer: MultimediaViewerNetworkPerformance schema validation errors - https://phabricator.wikimedia.org/T134165#2256447 (Gilles) [13:05:34] GOOD MORNNIIIIG [13:05:50] Hullo [13:06:06] holaaa [13:07:34] Analytics-Kanban, Operations, Patch-For-Review: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#2256496 (elukey) Websites still working since we had a problem with the VCL change, the following patch should fix the issue: https://gerrit.wikimedia.org/r/#/c/286419/ [13:13:06] elukey: stat1001 happening now? [13:14:43] ottomata: yeah in a bit, Brandon just merged https://gerrit.wikimedia.org/r/#/c/285976/2 since the original patch was missing the if in VCL (so it was a no-op) [13:15:25] aye cool [13:21:39] Analytics, RESTBase, Services, User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2256566 (JAllemandou) After weekend thinking: I don't there is any misunderstanding in the POV I describe :) Whether it is dec... [13:21:51] Analytics-EventLogging, MediaWiki-extensions-MultimediaViewer: MultimediaViewerNetworkPerformance schema validation errors - https://phabricator.wikimedia.org/T134165#2256567 (Gilles) p:Triage>Normal [13:24:32] (PS1) Mforns: Fix issues in metrics-by-project breakdown [analytics/dashiki] - https://gerrit.wikimedia.org/r/286424 (https://phabricator.wikimedia.org/T133944) [13:24:47] Analytics-Kanban, Patch-For-Review: Fix Dashiki's metrics-by-project breakdown - https://phabricator.wikimedia.org/T133944#2256572 (mforns) [13:24:50] Analytics-Kanban, Patch-For-Review: Visualize unique devices data in dashiki {bear} - https://phabricator.wikimedia.org/T122533#2256573 (mforns) [13:25:56] all right re-imagining stat1001 [13:27:18] (CR) Mforns: Fix issues in metrics-by-project breakdown (1 comment) [analytics/dashiki] - https://gerrit.wikimedia.org/r/286424 (https://phabricator.wikimedia.org/T133944) (owner: Mforns) [13:30:36] ottomata: I failed one thing, namely checking netboot.cfg.. stat1001 is not there :P [13:31:01] oh haha [13:31:12] dammit [13:31:21] welp! you'd need to change that to Jessie anyway [13:31:53] partman receipe you mean? [13:34:59] elukey: naw if you want to tell it to use the jessie installer during pxe boot [13:35:00] you need to tell it [13:35:04] unless jessie is now the default? [13:35:05] maybe it is [13:35:18] oh sorry, netboot is partman [13:35:19] sorry [13:35:22] was thinking of the host entries [13:36:17] nono I made jessie the defaul [13:36:20] *default [13:39:31] so from what I can see we have /dev/sda1 and a PV on /dev/sda2 (/srv) PV /dev/sda2 VG stat1001 lvm2 [5.43 TiB / 882.82 GiB free] [13:39:43] and I suspect phisical raid? [13:43:04] yeppa, megacli confirms [13:44:22] aye sounds right [13:45:47] ottomata: maybe I could re-image keeping /dev/sda2 untouched (flagging it in the installer) and then format only /dev/sda1? Then we'll need to create a proper partman receipe [13:47:55] sounds good elukey [13:48:31] two physical partitions each with a PV VG and LV doesn't sound too hard... [13:48:32] going to try, let's see what happens [13:48:35] k [13:48:56] but, ja, it would be hard to create a 'proper parman recipe' without testing it over and over during an install :/ [13:49:22] ouch.. so maybe it is better to try it now? [13:50:43] naw [13:50:49] i think it isn't worth it [13:50:54] it is only a single node [13:51:04] its not like you'll be applying any partman recipe you make here over and over again [13:51:28] the amount of effort to manually partition once every 3 or 4 years is WAY less than writing a partman recipe for use once every 3 or 4 years [13:58:29] ok done let's see if it works [14:01:56] Quarry, AutoWikiBrowser, WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256680 (Reedy) I suspect column number isn't the most user friendly, specifying the actual could be more useful So from https://quarry.wmflabs.org/run/84084/output/0/json-li... [14:02:50] Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2256684 (Ottomata) Hm, we were planning on running 2 cassandra instances per node for a total of 6 instances. Just stating the obvious here for my own benefit: - If we go with RAID 0... [14:03:34] Quarry, AutoWikiBrowser, WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2255546 (yuvipanda) You can also get the latest output for any query with the following URL format: `/query//result/latest//` [14:10:37] Quarry, AutoWikiBrowser, WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256705 (IKhitron) Thank you that you are interested. About column name - I thought about this, but sometimes it's not a good choice, because column name can be something as... [14:11:20] ottomata, elukey: I'll restart zookeeper on conf1* to pick up the new openjdk or is it a bad time? [14:11:51] moritzm: nope, nothing against it [14:12:58] moritzm: ja one at a time and it should be just fine [14:14:03] ottomata: sure, one at a time. [14:14:22] I also just realised that zookeeper isn't documented at https://wikitech.wikimedia.org/wiki/Service_restarts, will fix that later [14:14:56] moritzm: yep we'll need to fix a bit the docs, writing the node down.. also ottomata, do we have a dashboard for zookeeper? [14:15:07] I didn't find one in Grafana, we could build one [14:15:17] (we could == I am offering volunteer) [14:16:38] elukey: cool! no i don't thikn we have one [14:23:05] Quarry, AutoWikiBrowser, WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256735 (Reedy) Teaching a little SQL seems easier... ``` concat((case page_namespace when 0 then "" when 4 then "Project" when 14 then ":Category" when 118 then "Draft" when... [14:25:02] Quarry, AutoWikiBrowser, WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2256736 (IKhitron) I knew about AS, of course. I just wanted make it possible usage of existing queries without any need to change them. [14:33:49] ottomata: puppet fails for geowiki git credentials like sta1003 (https://phabricator.wikimedia.org/T132445), but I don't recall what you did to fix it [14:34:06] I added the ssh key to gerrit for stat1003 but you mentioned another procedure [14:36:01] looking [14:36:09] wha again!? [14:36:16] hola! [14:36:23] elukey: on stat1001? [14:36:26] or stat1003? [14:36:47] nuria_: hola! [14:37:14] ottomata: stat1001! [14:37:21] Analytics-Wikimetrics, Community-Wikimetrics: Make Wikimedia project titles more user-friendly - https://phabricator.wikimedia.org/T100692#2256773 (Zppix) p:Triage>Low [14:37:45] Error: /Stage[main]/Geowiki/Git::Clone[geowiki-scripts]/Exec[git_pull_geowiki-scripts]/returns: change from notrun to 0 failed: /usr/bin/git pull --quiet returned 1 instead [14:37:49] ottomata--^ [14:38:11] this one is a different repo though (it seems to me), so you might need to do the trick again? [14:38:23] (also let me know how you do it :P) [14:38:26] nuria_: holaa [14:41:23] elukey: the trick was just to fix the permissions/email for the user in gerrit [14:41:26] it should have been a one time fix [14:41:33] this looks different, it is pulling, not pushing [14:41:39] checking [14:42:22] ah yes but the last time it was also failing while pulling in stat1003 [14:42:29] and it was fixed adding the SSH key [14:42:31] in gerrit [14:42:45] the ssh key? [14:42:57] it shouldn't need an ssh key to pull [14:43:06] its pulling via the http addy [14:43:23] I am telling you what I remember from last time :) [14:43:38] :) [14:43:38] can I run puppet on stat1001? [14:43:50] sure sure [14:44:54] elukey: i see Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class passwords::statistics::user for stat1001.eqiad.wmnet on node stat1001.eqiad.wmnet [14:47:13] this is weird, I didn't get it the last time [14:47:15] let me try [14:48:30] ja tha'ts weird because that class def exists on palladium in private [14:48:45] ok, now i get the git pull error [14:49:03] hmm, looks like a weird perm error [14:49:04] Notice: /Stage[main]/Geowiki/Git::Clone[geowiki-scripts]/Exec[git_pull_geowiki-scripts]/returns: error: cannot open .git/FETCH_HEAD: Permission denied [14:49:35] hmm, i think the geowiki scripts clone happened before the stats user existed maybe... [14:49:37] not certain [14:50:22] elukey: , i just did [14:50:34] yeah I just chmod too [14:50:38] sudo chown -R stats:stats [14:50:40] haha [14:50:54] for /srv/geowiki/scripts and /srv/geowiki/scripts/.git [14:51:37] super weird, it was root:root [14:51:39] mmm [14:51:46] still not working? hm [14:51:54] .git/FETCH_HEAD: Permission denied [14:51:56] oh data private now [14:52:05] yeppa [14:52:15] doing chown... [14:52:16] group is www-data [14:52:53] oh whoops k [14:53:34] ok puppet is happier now [14:55:11] yepp.. I wonder if the permissions under /srv are correct now [14:58:46] zookeeper restart completed [14:59:01] \o/ [15:06:52] Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2256909 (Eevans) >>! In T133785#2256684, @Ottomata wrote: > Hm, we were planning on running 2 cassandra instances per node for a total of 6 instances. > > Just stating the obvious her... [15:20:46] so for some reason I have IRC issues [15:20:48] grrr [15:21:57] Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2256927 (Ottomata) > I assume you mean s/partition/array/ above. Indeed danke. > It was my understanding that the hardware order was informed by a prior decision to cluster 3 machines w... [15:32:46] ottomata: there is a problem with the base debian apache 2.4 config and the old one.. [15:32:55] oh ja? [15:33:22] basically the httpd.conf uses "Require" meanwhile the site-enabled ones the old Order deny bla bla [15:33:27] they don't play well together [15:33:43] especially if mod_authz_host (require) is set for / [15:33:54] soooo I would be inclined to fix the apache configs [15:34:01] only for 2.4 [15:34:22] elukey_: sounds good [15:34:29] all right sending the CR [15:34:39] i'm actually not very familiar with these. I'm sure i've touched them, but I did not create them :) [15:37:19] yeah the main suggestion from the httpd guys is to use either Order bla bla (mod_compact) or Require (mod_authz_hosts) [15:43:10] ottomata: https://gerrit.wikimedia.org/r/#/c/286461/1 [15:44:26] cool elukey_ makes sense, you tested/should I just merge? [15:45:32] ottomata: I tested it manually on the host and it works, lemme check on thing to be sure [15:45:46] Analytics, Analytics-Cluster: Puppetize and deploy MirrorMaker using confluent packages - https://phabricator.wikimedia.org/T134184#2256991 (Ottomata) [15:46:06] elukey_: that sounds right, cool, if tested manually on host [15:46:17] elukey_: +2 i'll let you submit merge [15:46:48] ottomata: yep.. my only doubt is that there are also other Requires [15:46:54] that is weird [15:47:23] I mean, I just noticed that there are for example Require user wmf [15:52:43] nuria_: will join the ops meeting and send the escrum later, was busy with stat1001 :( [15:53:20] elukey_: k, let us know when outage is over [15:53:23] i'm going to come to standup and ask to go first, then will join ops meeitng :) [15:58:05] ottomata: merged! now curl localhost return me the default vhost, namely datasets [15:58:24] I updated the code review removing Allow directives since they were used for stats together with require [15:58:42] doesn't make sense to add password protected stuff and then allow all right? [15:59:10] elukey_: i guess so! :p [16:00:56] also NameVirtualHost is deprecated [16:00:57] arghh [16:00:58] a-team standup! [16:01:02] whaa?! [16:01:49] ottomata: yep doesn't make sense anymore for 2.4 [16:03:25] "The NameVirtualHost directive no longer has any effect, other than to emit a warning. Any address/port combination appearing in multiple virtual hosts is implicitly treated as a name-based virtual host." [16:07:36] oh ja elukey_ btw, do you know anything about that analytics1001 namenode flap? [16:08:06] nope, I saw that it was among tons of other ones and I didn't care too much (it recovered right afterwards IIRC) [16:08:10] ottomata: --^ [16:08:29] yeah, same. [16:08:31] hm [16:08:52] Analytics-Kanban: Create maintenance page for stat1001 maintenance - https://phabricator.wikimedia.org/T133401#2257061 (mforns) [16:08:56] Analytics-Kanban: Create maintenance page for stat1001 maintenance - https://phabricator.wikimedia.org/T133401#2257062 (Nuria) Open>declined [16:09:04] Quarry, AutoWikiBrowser, WorkType-Maintenance: Quarry run result in AWB make list - https://phabricator.wikimedia.org/T134141#2257063 (IKhitron) To explain you, what did I mean about encoding. When I run Quarry, I never can just use the results. I open the CSV download in Notepad++, convert it to UTF... [16:12:30] ottomata: curl localhost/reportcard/staff -H "Host: stats.wikimedia.org" says (correctly) unauthorized following httpd config [16:12:34] all websites are working [16:13:23] great! [16:14:14] so after the ops meeting I'll revert the varnish change [16:14:15] :) [16:15:14] awesooome! [16:20:21] (PS1) Joal: Add script allowing rerunning oozie jobs [analytics/refinery] - https://gerrit.wikimedia.org/r/286471 [16:21:49] mforns: on your dashiki patch about patterns... there are no breakdowns on wikimetrics metrics correct? How do i repro the bug? [16:22:57] * joal is not going to rerun every oozie job manually EVER !!! [16:23:21] (CR) Ottomata: [C: 1] Include webrequest refine oozie job into load one [analytics/refinery] - https://gerrit.wikimedia.org/r/285998 (https://phabricator.wikimedia.org/T130731) (owner: Joal) [16:26:22] joal: you keep saying that : [16:26:23] :P [16:27:06] I'll use my script even if you don't +2 it elukey :-PP [16:27:19] ahahahahhhaah [16:27:30] * elukey hugs joal [16:27:36] (CR) Nuria: "Will be great to document this on our oozie page in wikitech:" [analytics/refinery] - https://gerrit.wikimedia.org/r/286471 (owner: Joal) [16:27:45] OH cool joal this uses the REST API? [16:27:52] ottomata: it does [16:28:07] nuria_: Yes M'dame, will do [16:28:12] cool [16:28:34] joal: jaja ok! [16:29:06] (CR) Ottomata: "Nice, looks good. A couple of nits." (2 comments) [analytics/refinery] - https://gerrit.wikimedia.org/r/286471 (owner: Joal) [16:29:55] ottomata: Have there been varnish issues/restarts today ? [16:30:13] Analytics: ==== Immediate Above ==== - https://phabricator.wikimedia.org/T115634#2257122 (Nuria) [16:30:14] ottomata: text and upload load jobs have issues for hout 14 :( [16:30:26] joal: esams went offline [16:30:33] due to MPLS issues [16:30:34] a-team: groomingggg!!! [16:30:37] elukey: RIIIIIIGHT [16:31:07] elukey: There is nothing we can do then :( [16:31:15] :( [16:32:23] Analytics-Cluster, Analytics-Kanban: Puppetize and deploy MirrorMaker using confluent packages - https://phabricator.wikimedia.org/T134184#2257131 (Nuria) [16:33:48] Analytics: Make a script to automatise the 4 commands to run for aqs deployment - https://phabricator.wikimedia.org/T133863#2257138 (Nuria) [16:34:44] joal: just to double check, hour 14 in which timezone? UTC? [16:34:53] yessir [16:35:07] if so it would be ~2 hrs ago, so yeah it was esams :) [16:35:23] okey elukey :) [16:35:28] Thanks for confirmation [16:38:00] Analytics, RESTBase: REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245#2257155 (Nuria) >I am not aware of any general real-time traffic analytics ability in Varnish, and I think there are good architectural reasons for minimizing such processing... [16:38:10] Analytics, RESTBase: REST API entry point web request statistics at the Varnish level - https://phabricator.wikimedia.org/T122245#2257156 (Nuria) [16:38:57] Analytics, Analytics-Cluster, Patch-For-Review: https://yarn.wikimedia.org/cluster/scheduler should be behind ldap - https://phabricator.wikimedia.org/T116192#2257162 (Nuria) [16:40:35] mforns: whenever you have a minute can you double check if the dashboards relying on stat1001's data are working fine? [16:40:44] Analytics: Polish script that checks eventlogging lag to use it for alarming - https://phabricator.wikimedia.org/T124306#2257169 (Nuria) [16:40:47] elukey, sure [16:40:51] \o/ [16:41:27] Analytics, Operations, Security, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/api - https://phabricator.wikimedia.org/T92338#2257172 (Nuria) [16:41:35] Analytics, Operations, Security, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/glam_nara - https://phabricator.wikimedia.org/T92340#2257175 (Nuria) [16:41:40] Analytics, Operations, Zero, Mobile, and 2 others: Purge > 90 days stat1002:/a/squid/archive/mobile - https://phabricator.wikimedia.org/T92341#2257177 (Nuria) [16:41:44] Analytics, Operations, Zero, Security, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/sampled - https://phabricator.wikimedia.org/T92342#2257179 (Nuria) [16:41:47] Analytics, Operations, Zero, Security, audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/zero - https://phabricator.wikimedia.org/T92343#2257181 (Nuria) [16:43:02] Analytics, Pageviews-API: Enable pageviews on nl/be.wikimedia.org {melc} - https://phabricator.wikimedia.org/T127804#2257184 (Nuria) [16:43:48] Analytics, Analytics-Cluster: Audit kernel version on analytics worker nodes - https://phabricator.wikimedia.org/T109834#2257186 (Nuria) [16:44:21] Analytics: Host a debrief of EventLogging cleanup {tick} - https://phabricator.wikimedia.org/T104351#2257187 (Nuria) Open>Resolved [16:44:23] Analytics-EventLogging, Analytics-Kanban: {tick} Schema Audit - https://phabricator.wikimedia.org/T102224#2257188 (Nuria) [16:45:11] Analytics: Create new table for 'referer' aggregated data - https://phabricator.wikimedia.org/T112284#2257189 (Nuria) [16:46:25] Analytics, Analytics-EventLogging: Upgrade eventlogging servers to Jessie - https://phabricator.wikimedia.org/T114199#2257192 (Nuria) [16:50:36] Analytics, Continuous-Integration-Infrastructure, Patch-For-Review: Add json linting test for schemas in mediawiki/event-schemas - https://phabricator.wikimedia.org/T124319#2257219 (Nuria) [16:50:49] Analytics: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#2257222 (Nuria) [16:51:50] Analytics, Pageviews-API: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#1929942 (Nuria) [16:52:50] Analytics, Pageviews-API: Pageview API: Better filtering of bot traffic on top enpoints - https://phabricator.wikimedia.org/T123442#2257234 (Nuria) [16:54:14] Analytics: analytics specific icinga alerts should ping in our IRC channel. - https://phabricator.wikimedia.org/T125128#2257236 (Nuria) [16:54:46] Analytics, WMDE-Analytics-Engineering: Remove http://datasets.wikimedia.org/aggregate-datasets/wikidata/ - https://phabricator.wikimedia.org/T125407#2257237 (Nuria) [16:55:11] Analytics: Consider SSTable bulk loading for AQS imports - https://phabricator.wikimedia.org/T126243#2257238 (Nuria) [16:55:42] got signed out a-team, logging back on [16:56:02] Analytics: Create fake data for beta AQS deployment - https://phabricator.wikimedia.org/T120841#2257240 (Nuria) [16:56:27] Analytics: Create fake data for beta AQS deployment - https://phabricator.wikimedia.org/T120841#1862546 (Nuria) [16:57:02] elukey, seems that the dashboards have CORS problems when accessing the datasets [16:57:21] mforns: ouch [16:57:30] :S [16:57:47] Analytics: Compile a request data set for caching research and tuning - https://phabricator.wikimedia.org/T128132#2257249 (Nuria) [16:59:02] elukey, do you have any idea why this happens? [16:59:23] mforns: do you mind if we batcave? [16:59:40] elukey, sure! omw [17:01:52] a-team, nuria_. I will batcave for a while with Luca to try to know why dashiki dashboards are now with CORS problems [17:01:53] joal, ottomata let's met on batcave [17:02:14] mforns: CORS? whats up [17:03:10] madhuvishy, we are in the batcave with Luca if you want to join [17:03:25] after 1001 changes, dashboards have CORS problems [17:10:28] the main problem for the CORS issues is Luca [17:10:40] because of course now datasets.w.o returns 503 to all [17:10:48] sorry guys a bit tired :P [17:10:58] I am going to revert the Varnish 503 change [17:33:13] !log reverted Varnish config to return 503s for datasets and stats [17:33:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [17:38:47] !log removed out of service banner from dashiki dashboards [17:38:49] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [17:44:20] a-team: I can browse in datasets and stats without any problem, and the dashboards seem ok.. would you mind to double check for weird things? [17:44:40] elukey, on it [17:50:27] elukey, everything seems OK! [17:50:33] woooo [17:50:36] :) [17:50:45] thanks a lot mforns! [17:52:05] I found that the dashiki breakdown bug (that I was fixing today), not only broke dygraphs in the metrics-by-project, but in the other layouts, too... so the browser reports are broken, and all other dashboards that use the dygraphs line chart [17:52:09] nuria_, ^ [17:52:48] I'll have to check if the fix that is in CR also fixes the problems for the other layouts, and my guess is no [17:54:24] :/ [17:56:32] all right, so I am going to log-off, but I'll double check in a bit if any issue arise with stat1001 [17:56:35] byeeeee [17:57:08] elukey, bye! [18:14:56] mforns_afk: arghhh [18:15:14] mforns_afk: should we roll back the last push? [18:30:21] !log manually touch _SUCCESS file in hdfs://analytics-hadoop/wmf/data/raw/webrequest/webrequest_text/hourly/2016/05/02/14/ to launch refine process despites load job failure [18:30:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master [18:43:49] hey kevinator. is unsubscribing Reguyla from Analytics something you can help with? [18:44:13] yes, I can do that [18:44:20] I was about to and saw Neils email... [18:44:27] that would be great. You will save the current thread kevinator. thanks. [18:44:32] now reading reguyla's email [18:45:23] he could just email the list admins, but he doesn't, kevinator. It's easier to remove him from the list per his request and move on. [18:47:32] done [18:47:47] thanks, kevinator [18:47:56] heading out for lunch, see y'all later. [18:48:05] ciao [18:50:50] Analytics-Cluster, Analytics-Kanban: Publicize stat1004 - https://phabricator.wikimedia.org/T133056#2257457 (Ottomata) [19:14:15] mforns_afk: working on fix [19:14:52] a-team I'm gone for tonight [19:14:56] laters! [19:44:13] woo stat1004! [20:23:50] nuria_, I'm here [20:24:01] mforns: trying to fix this, messy mess [20:24:13] mforns, I can rollback [20:24:30] mforns: i wish i had realized this earlier, it is so obvious that I do not know why didn't i see it [20:24:33] O.o I sent a message to myself... [20:24:37] jaja [20:25:00] nuria_, do you want me to rollback? [20:25:01] let's rollback only the patterns stuff and leave the outage box that has no issue [20:25:26] mforns: I think we have to cause i though i could fix it well in couple hours but no, it will take a bitmore [20:25:26] nuria_, ok will do [20:25:46] mforns: i can do it too, let's do it together, let me see if revert in gerrit works clean [20:25:59] nuria_, yes, I was looking at it and I know what happens, I think I can fix it today [20:26:04] nuria_, batcave? [20:26:24] mforns: i looked at it too, but it is not a matter of adding a ko.computed [20:26:54] mforns: we can batcave [20:27:09] nuria_, omw [21:11:04] (PS2) Mforns: Fix issues in metrics-by-project breakdown [analytics/dashiki] - https://gerrit.wikimedia.org/r/286424 (https://phabricator.wikimedia.org/T133944) [21:12:51] (CR) Mforns: [C: 2 V: 2] "Self-merging to unbreak production :/" [analytics/dashiki] - https://gerrit.wikimedia.org/r/286424 (https://phabricator.wikimedia.org/T133944) (owner: Mforns) [21:18:50] !log deployed browser-reports to unbreak production [21:35:54] !log deployed language-reportcard [21:59:11] !log deployed edit-analysis [21:59:27] !log deployed vital-signs [22:20:19] Analytics, MediaWiki-API, Pageviews-API, RESTBase, RfC: RFC: Update profile URLs in content types to point to format documentation - https://phabricator.wikimedia.org/T128609#2258014 (Danny_B)