[00:03:20] (03Abandoned) 10EBernhardson: Support hourly or daily partition dropping [analytics/refinery] - 10https://gerrit.wikimedia.org/r/419949 (https://phabricator.wikimedia.org/T189845) (owner: 10EBernhardson) [00:16:11] (03PS1) 10EBernhardson: Drop hive partitions before checking hdfs paths [analytics/refinery] - 10https://gerrit.wikimedia.org/r/419953 [00:25:37] (03CR) 10Zhuyifei1999: [C: 032] worker: raise the save time limit to 10 mins [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/419786 (https://phabricator.wikimedia.org/T188564) (owner: 10Zhuyifei1999) [00:25:49] (03Merged) 10jenkins-bot: worker: raise the save time limit to 10 mins [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/419786 (https://phabricator.wikimedia.org/T188564) (owner: 10Zhuyifei1999) [00:46:30] (03PS1) 10Sturmkrahe: Darken "subdued" text and lighten footer text for WCAG AA compliance [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419958 (https://phabricator.wikimedia.org/T185533) [00:52:32] 10Analytics: Mount dumps on SWAP machines (notebook1001.eqiad.wmnet / notebook1002.eqiad.wmnet) - https://phabricator.wikimedia.org/T176091#4055406 (10madhuvishy) [00:53:57] 10Analytics: Mount dumps on SWAP machines (notebook1001.eqiad.wmnet / notebook1002.eqiad.wmnet) - https://phabricator.wikimedia.org/T176091#3613231 (10madhuvishy) I think the firewall stuff already exists now, we can do this as part of the dumps NFS migration on April 2, and have the shares available from labsto... [01:31:26] (03PS1) 10Mrcornacchio: Change TopicExplorer placeholder contrast to WCAG AA compliance [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419962 (https://phabricator.wikimedia.org/T185533) [02:10:29] 10Quarry, 10Patch-For-Review: Quarry should refuse to save results that are way too large - https://phabricator.wikimedia.org/T188564#4055499 (10zhuyifei1999) https://quarry.wmflabs.org/query/25563 didn't get killed :( [06:35:47] (03PS1) 10Sahil505: The alert message about adblocker is not fully shown on smaller screens [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) [06:41:58] 10Analytics: Mount dumps on SWAP machines (notebook1001.eqiad.wmnet / notebook1002.eqiad.wmnet) - https://phabricator.wikimedia.org/T176091#4055615 (10elukey) >>! In T176091#4054301, @Ottomata wrote: > @elukey, can we also add a hole in the analytics vlan firewall for this? I'd like to mount dataset dumps on th... [07:04:28] 10Analytics, 10Analytics-Wikistats: Beta: Y-axis units and rounding issues - https://phabricator.wikimedia.org/T187429#4055624 (10sahil505) Hey, we know now that we can't place the legend (with the date and the values of the point) anywhere over the line graph as it hinders with the hovering and also makes the... [07:06:00] (03CR) 10VolkerE: Change TopicExplorer placeholder contrast to WCAG AA compliance (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419962 (https://phabricator.wikimedia.org/T185533) (owner: 10Mrcornacchio) [08:17:43] analytics1077 deployed, last worker node to be added [08:17:49] Yay elukey ! [08:17:56] Thanks for MOAR ! [08:18:08] still fighting a bit with puppet dependencies, the first puppet run wasn't super clean [08:18:43] * joal wonders if he could even think of a puppet run as something clean [08:18:48] * joal hides [08:20:50] * elukey throws puppets to joal [08:21:21] * joal hides deeper in his coffee mug [08:21:22] I seriously don't get dependency management in puppet [08:21:29] and the docs are really not clear [08:21:50] elukey: Do you want to discuss in batcave? I could do what I do best (rubberducking ...) [08:23:30] joal: don't worry I just sent a code review to Andrew that should fix the issue, but it adds explicit ordering between classes that I thought could be avoidable simply using require [08:23:39] but from the puppet run logs it is clearly not the case [08:24:00] so I am pretty sure that there are weird interactions that I still don't know :) [08:24:21] elukey: I'm assuming that sometimes running different things in order is important - How that is achievable in puppet, I can't say [08:24:25] :) [08:24:36] me too :P [08:24:46] to continue the fun, let's reboot druid nodes! [08:25:01] do you have some time to follow my ramblings? [08:25:16] (just reading and checking that druid behaves) [08:25:20] elukey: I always have time to talk with you :) [08:25:34] * elukey sends wikilove to joal [08:25:38] elukey: nothing noticeable yesterday on my side on druid reboots :) [08:26:51] joal: have you ever watched the tv series "Flash Forward" ? [08:27:28] nope elukey - With kids home, I can't even recall what was the last I watched an entire movie without pauses [08:28:03] ah it was a old one, several years ago [08:28:14] ah - Icould have then - but nope :) [08:28:41] nothing then, it was to make a joke about foreseeing fires in the future when saying "nothing weird registered, let's go ahead" [08:29:13] Arf - Sorry for being accultured :) [08:29:49] hahahahahah [08:30:08] all right druid1005 removed from lvs and silenced, stopping daemons [08:34:05] joal: did you see https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ ? [08:34:14] elukey: I have yes [08:34:47] elukey: I really think clickhouse is awesome - I however think it requires a good bit of work to automate ingestion [08:35:14] "Kafka DNS topic has on average 1.5M messages per second vs 6M messages per second for HTTP requests topic." [08:36:26] druid1005 is up [08:37:03] re-pooled, checking some broker queries [08:38:27] seems fine.. [08:39:22] overlord console on druid1004 looks good (all middlemanagers registered) [08:42:13] doing 1006 [08:43:43] elukey: interesting shape of caching hits :) [08:47:16] 1006 up and running [09:57:20] !log restart eventlogging-consumer@mysql-m4/eventbus on eventlog1002 to force the DNS resolution of m4-master (changed from dbproxy1009 -> dbproxy1004) [09:57:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:58:15] about --^: just restarted the mysql consumers on eventlog1002 to force the dns resolution of m4-master.eqiad.wmnet [09:58:25] it is a CNAME to a dbproxy, that in turn points to db1107 [09:58:34] (dba maintenance in progress) [09:58:39] if you see anything weird let me know [11:33:33] * elukey lunch + errand! Be back in ~2h [11:48:10] (03PS3) 10Joal: [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) [11:50:52] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) (owner: 10Joal) [12:04:26] hey teaaaam :] [12:05:16] Heya mforns :) [12:05:29] hi joal! [12:10:30] 10Analytics, 10Analytics-Wikistats: Change '--' to something more helpful in Wikistats page views by coutry table view - https://phabricator.wikimedia.org/T187427#4056155 (10mforns) Hi @Sahil :] I think "Unknown" would be the best. Would that sound OK to you? [12:15:26] 10Analytics, 10Analytics-Wikistats: Change '--' to something more helpful in Wikistats page views by coutry table view - https://phabricator.wikimedia.org/T187427#4056162 (10sahil505) I was thinking the same :) Plus I don't think that we have any other reasonably good option besides that. [12:18:05] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: The alert message about adblocker is not fully shown on smaller screens - https://phabricator.wikimedia.org/T188208#4056164 (10mforns) @sahil505, thanks for taking care of this task! Your fix looks good to me! Maybe, now that we're using 2 lines, I woul... [12:18:28] 10Analytics, 10Analytics-Wikistats: Change '--' to something more helpful in Wikistats page views by coutry table view - https://phabricator.wikimedia.org/T187427#3974857 (10JAllemandou) Hi lads - The reason for which we used '--' in compuing stats was because of https://en.wikipedia.org/wiki/Unknown I'm super... [12:24:53] 10Analytics, 10Analytics-Wikistats, 10Patch-For-Review: The alert message about adblocker is not fully shown on smaller screens - https://phabricator.wikimedia.org/T188208#4056171 (10sahil505) Thanks @mforns. I agree about the line break, will make it look a bit more uniform and cleaner. On it! [12:31:53] (03CR) 10Mforns: "Thanks for fixing this Sahil! I left a couple comment in the commit message. Cheers!" (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) (owner: 10Sahil505) [12:40:40] 10Analytics, 10Analytics-EventLogging, 10Readers-Web-Backlog: Should it be possible for a schema to override DNT in exceptional circumstances? - https://phabricator.wikimedia.org/T187277#4056194 (10phuedx) [12:48:07] 10Analytics, 10Analytics-Wikistats, 10Accessibility, 10Easy, 10Patch-For-Review: Wikistats Beta: Fix accessibility/markup issues of Wikistats 2.0 - https://phabricator.wikimedia.org/T185533#4056220 (10mforns) @sahil505 I think we should check both. See that there's already one Gerrit patch to fix that pa... [13:16:40] (03PS2) 10Sahil505: Fixed the adblocker alert message overflow for smaller screens [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) [13:18:24] (03PS3) 10Sahil505: Fixed the adblocker alert message overflow for smaller screens [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) [13:21:04] (03CR) 10Sahil505: "Thanks for reviewing this Marcel :) Will try my best in future to properly name my commit messages. I have added a line break in the alert" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) (owner: 10Sahil505) [13:27:04] 10Analytics, 10Analytics-Wikistats: Beta: Y-axis units and rounding issues - https://phabricator.wikimedia.org/T187429#4056327 (10mforns) @sahil505 Yes, I agree this is an option. The majority of single-line charts have their line travel through the top section of the canvas. We could move the popup-legend dow... [13:40:18] 10Analytics, 10Analytics-Wikistats: Beta: Y-axis units and rounding issues - https://phabricator.wikimedia.org/T187429#4056386 (10sahil505) @mforns This is interesting. Will try my best. Whatever option we go with it should be more convenient to the user to relate to. Maybe want to keep this for the GSoC codin... [13:42:12] all right so the two druid boxes left to reboot are druid1001/1002 [13:43:24] !log disable druid1002's middle manager via API as prep step for reboot [13:43:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:43:36] elukey: openjdk-8 update is finally out, I'm currently building packages for jessie-wikimedia [13:44:04] * elukey cries in a corner [13:44:10] (03CR) 10Ottomata: "Hm I think +1 ,but could be tricky. I collected both partition and paths to remove first, so that if there were any exceptions thrown dur" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/419953 (owner: 10EBernhardson) [13:44:19] does it mean rolling restart of everything?? [13:44:30] yarrrr [13:44:35] elukey: let's upgrade to stretch [13:44:40] as part of rolling restart :p [13:44:44] updates for java 7 are not yet out, so "only" for the stuff running 8 [13:44:53] moritzm: everything is java 8 now! :p [13:44:59] druid isn't? [13:46:11] yeah druid isn't yet, but it will be soon :D [13:46:29] see, so the current reboots aren't wasted :-) [13:47:10] ottomata: o/ - let's wait a bit more to see if anything medium-long term comes up with stretch on analytics107* :D [13:47:36] it could be a good occasion though to rollout the new change for the datanodes (resiliency to 2/3 volumes failures) [13:49:23] +1 [13:54:57] elukey: FYI jumbo mirror maker is still flaky [13:55:04] restarted it several times yesterday, just restarted one of htem now [13:55:36] i might put eventlogging eventbus mysql consumer back on analytics, since that mirror maker isworking mostly fine [13:55:46] and then experiement next week with settings and or other options like ureplicator [14:00:22] ottomata: ouch sorry didn't check today :( [14:00:41] +1, let me know If I can help [14:04:42] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Provide Hatstall to fix syncing / updating of identities on the wikimedia.biterg.io production instance - https://phabricator.wikimedia.org/T157898#4056440 (10Aklapper) Tentative plan for this to go live is April, according to https://gi... [14:24:59] !log changed superset druid private config from druid1002 to druid1003 [14:25:00] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:26:34] druid1002 is rebooting [14:32:19] ottomata: I'd need to reboot kafka100[23] but I can wait next week for the eventbus mysql consumer changes [14:33:01] elukey: ok will do that now [14:33:25] elukey: i think it will start of from where it left off when we switched it earlier this week [14:33:26] ottomata: nono you can do it next week without any rush, I was just asking :D [14:33:26] so we [14:33:30] we'll get those duplicates again [14:33:33] ah okok [14:33:33] naw now is good [14:33:38] :) [14:33:39] but not as many as before :) [14:34:16] ottomata: as FYI today the dbas changed the m4-master cname from dbproxy1009 to 1004, and I had to restart the mysql consumers to pick up that change [14:34:30] all good and logged, but just wanted to report it :) [14:34:40] 10Analytics, 10EventBus, 10Services (next): Support multiple partitions per topic in EventBus - https://phabricator.wikimedia.org/T157822#4056518 (10mobrovac) [14:34:45] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Services (designing): Support per-db-shard concurrency in ChangeProp - https://phabricator.wikimedia.org/T189738#4056517 (10mobrovac) [14:37:54] !log disable druid1001's middlemanager as prep step for reboot [14:37:55] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:38:11] !log temporary point pivot to druid1002 as prep step for druid1001's reboot [14:38:11] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:38:21] ok [14:44:00] !log restarting eventlogging mysql eventbus consumer to consume from analytics instead of jumbo [14:44:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:51:19] (03CR) 10Mforns: "The fix looks really good :]" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) (owner: 10Sahil505) [14:56:31] 10Analytics, 10Analytics-Wikistats: Beta: Y-axis units and rounding issues - https://phabricator.wikimedia.org/T187429#4056606 (10mforns) Sure @sahil505, please do take your time! None of those tasks was intended to be done before the coding period! It's great you're working on them, but please don't feel any... [15:03:57] 10Analytics, 10Analytics-Wikistats: Change '--' to something more helpful in Wikistats page views by coutry table view - https://phabricator.wikimedia.org/T187427#4056639 (10mforns) @JAllemandou I see. There's this problem with article names. But in this case, though, where we are dealing with country names, I... [15:17:18] druid1001 drained (finally), rebooting [15:20:17] 10Analytics-Kanban, 10User-Elukey: Reboot all Analytics hosts for Kernel upgrade - https://phabricator.wikimedia.org/T188594#4056691 (10elukey) [15:20:19] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Have "Last Attracted Developers" information for Gerrit automatically updated / Integrate new demography panels in GrimoireLab product - https://phabricator.wikimedia.org/T151161#4056693 (10Aklapper) [15:20:21] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Advertise wikimedia.biterg.io more widely in the Wikimedia community - https://phabricator.wikimedia.org/T179820#4056692 (10Aklapper) [15:20:32] 10Analytics-Kanban, 10User-Elukey: Reboot all Analytics hosts for Kernel upgrade - https://phabricator.wikimedia.org/T188594#4012985 (10elukey) [15:27:56] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): One account (in "gerrit_top_developers" widget) counted as two accounts (in "gerrit_main_numbers" widget) - https://phabricator.wikimedia.org/T184741#4056727 (10Aklapper) [15:36:02] so for the moment nothing weird has been registered on the druid private/analytics cluster related to overlord/middlemanager [15:36:11] this time I stopped zookeeper before the rest [15:36:13] on each node [15:36:29] and it seems not causing the weird race conditions that were happening during the past reboots [15:41:14] elukey: I'm adding thorium to https://phabricator.wikimedia.org/T188594, seems to have fallen through the cracks [15:41:31] 10Analytics-Kanban, 10User-Elukey: Reboot all Analytics hosts for Kernel upgrade - https://phabricator.wikimedia.org/T188594#4056755 (10MoritzMuehlenhoff) [15:41:37] moritzm: you are right, forgot about it! [15:41:39] thanks :) [15:45:22] 10Analytics: Create reports in wikistats for "most prolific editors" - https://phabricator.wikimedia.org/T189882#4056771 (10Nuria) [15:55:16] moritzm: qq for the cassandra22 component - in this is case is it sufficient to add it to distributions-wikimedia? [15:55:39] (possibly cassandra226) [15:57:21] yeah, adding it to distributions/wikimedia is all [15:57:49] after running puppet on install1002, you can then reprepro include the packages [15:58:03] nice! I'll start working on it on Monday [16:03:06] 10Analytics, 10Analytics-Kanban: [EL sanitization] Modify mysql purging script to read from the new YAML whitelist - https://phabricator.wikimedia.org/T189692#4056830 (10mforns) a:03mforns [16:06:54] 10Analytics-Kanban, 10Patch-For-Review: Remove sensitive fields from whitelist for QuickSurvey schemas (end of Q2) - https://phabricator.wikimedia.org/T174386#4056838 (10Nuria) @fdans to apply changes to historical data [16:26:42] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)), 10Patch-For-Review: Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#4056916 (10Jdlrobson) Is anything left to do here..... ? [16:26:54] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)): Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#4056917 (10Jdlrobson) [16:38:17] 10Analytics, 10Analytics-Wikistats: Remaining reports. - https://phabricator.wikimedia.org/T186121#4056967 (10Nuria) https://www.mediawiki.org/wiki/Analytics/Wikistats/TrafficReports/Future_per_report_B2 [16:42:21] joal: this time I stopped zookeeper first on druid100[123] nodes, and then followed this: broker, historical, middlemanager, overlord, coordinator. As far as I can see the real time indexation didn't flinch, so maybe simply stopping zk first is the key to avoid all those weird race conditions [16:42:57] elukey: I'm even surprised that everything works good with zk stopped ! [16:51:13] 10Analytics, 10Analytics-Wikistats: Remaining reports. - https://phabricator.wikimedia.org/T186121#4057017 (10Nuria) Some info from erik: Here is the second survey which deals with non traffic reports. https://www.mediawiki.org/wiki/Analytics/Wikistats/DumpReports/Future_per_report This survey was held in... [16:53:35] 10Analytics: Create reports in wikistats for "most prolific editors" - https://phabricator.wikimedia.org/T189882#4057021 (10Nuria) [16:53:38] 10Analytics, 10Analytics-Wikistats: Remaining reports. - https://phabricator.wikimedia.org/T186121#4057020 (10Nuria) [17:01:57] joal: one node at the time! :) [17:02:03] Au :) [17:02:05] Ah [17:02:16] * joal is dumb [17:02:33] yeah sure, if you are I am probably a vegetable :D [17:02:51] * joal imagines which veggie elukey could look like [17:02:57] :D [17:14:09] ottomata: hey I put up https://gerrit.wikimedia.org/r/#/c/420078/ to enable rsyncs to labstore1006/7 [17:14:41] +1ed [17:14:42] oh elukey sorry I didn't know you did too! [17:14:52] that's super fine! [17:14:56] feel free to merge it [17:15:08] :) thank you [17:15:37] do either of you have a sec to chat about the nfs mount from labstore* to stat? [17:16:11] madhuvishy: sure [17:17:00] ottomata: so now that we have two redundant servers, the plan is we shard dumps distribution usecases across them (nfs to vps and stat in one, and web + rsync mirrors in the other) [17:17:23] but i'm setting them up so that we can failover all services to either if needed for maint or emergencies [17:17:29] 10Analytics, 10Operations, 10Ops-Access-Requests, 10Research, and 2 others: Restricting access for a collaboration nearing completion - https://phabricator.wikimedia.org/T189341#4039520 (10RobH) a:03DarTar This seems to be awaiting @dartar to confirm that the list (initially including just @Michele.tizzo... [17:17:33] k [17:18:22] The NFS failover plan for VPS is, we mount from both servers at all times to a path in /mnt/, and the active server mount is symlinked to /public/dumps - which is the only path that our users use [17:18:56] I have a detailed plan written down here In stat that path is now /mnt/ [17:18:59] uhh [17:19:06] https://phabricator.wikimedia.org/T171540 [17:19:35] In stat that path that the share is mounted on now is /mnt/data [17:19:50] I'd like to do the same mechanism for stat too if that seems okay with you [17:20:10] proof of concept for VPS - https://gerrit.wikimedia.org/r/#/c/403767/ [17:22:27] madhuvishy: cool, so stat1006 would have both labstore1006 and 1007 mounted, but symlink for active at /data [17:22:28] ? [17:23:09] yup correct, the paths where they are mounted and symlinked are up to you, I think they are available currently at /mnt/data [17:23:09] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)): Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#4057123 (10mforns) @Jdlrobson > Is anything left to do here..... ? I don't think so! The schema is whitelisted, i... [17:23:22] oh you are right [17:23:24] /mnt/data [17:23:25] cool [17:23:28] ya best to keep it the same [17:23:32] 10Analytics, 10Operations, 10Ops-Access-Requests, 10Research, and 2 others: Restricting access for a collaboration nearing completion - https://phabricator.wikimedia.org/T189341#4057125 (10DarTar) @RobH correct, it should apply to all four users, thanks for catching this. [17:23:44] madhuvishy: in case you want to, (you might not care), update-alternatives was built for managing symlinks like this [17:23:46] right, any ideas for where you want the underlying mounts to go? [17:23:50] or you might just do it in puppet [17:23:58] madhuvishy: i'd say in /mnt as well [17:24:04] as you have them listed is fine [17:24:18] i don't have much of a preference [17:24:21] okay cool, I'll put up a patch then, we can merge when we actually migrate [17:24:24] k [17:24:42] yeah my only hesitation is people using the underlying mounts from the non-active server [17:25:02] oh because...how do you keep them in sync [17:25:08] rsync active -> standby? [17:25:10] cron? [17:25:26] both get rsyncs from source (dumpsdata1001/2) [17:25:31] * elukey logs off, byyyeee o/ [17:25:39] oh [17:25:40] it won't actually be perfectly in sync [17:25:43] byyyee [17:26:01] oh madhuvishy then i dunno, maybe it isn't a big deal to have them accessible, we tell folks not to use them directly [17:26:05] but if they do its ok, ya? [17:26:10] they are read-only, right? [17:26:31] yup, they are, just don't want to overload one server vs the other [17:26:42] cool, we can always revisit if this is a problem [17:28:53] thanks ottomata, I'll put up a patch, we can merge when the migration actually happens [17:30:29] k cool [17:32:17] ottomata: also I'm gonna turn on hdfs-archive rsyncs from stat1005 to labstore1006,7 on top of dataset1001 soon, to make sure they are syncing okay. Let me know if it causes any issues on your end. [17:33:32] ok cool [17:33:33] sounds good [17:33:41] to be specific, this https://github.com/wikimedia/puppet/blob/production/modules/dumps/manifests/web/fetches/stats.pp [17:33:45] cool thanks a ton! [17:34:37] (03PS4) 10Joal: [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) [17:36:34] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Update mediawiki-history spark job for performance [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/419516 (https://phabricator.wikimedia.org/T189449) (owner: 10Joal) [17:56:42] ottomata: I put up a patch here for the mounts https://gerrit.wikimedia.org/r/#/c/420083/ [17:57:00] I assume you can also apply it to notebook* once this is done [17:58:06] madhuvishy: ya we gotta get luca to make a vlan hole but ya [17:58:55] madhuvishy: +1, let's also add a README file in /mnt/nfs explaining those and telling folks to use /mnt/data instead [17:59:20] ottomata: I thought he already opened up all traffic to labstore1006&7 [18:00:40] sure that's a good idea [18:03:19] oh all traffic? oh ok [18:03:20] cool [18:03:26] makes sense riiight [18:03:27] ok cool [18:03:29] aewsome will do [18:10:54] ottomata: added a readme ;) [18:10:57] :) [18:12:05] 10Analytics, 10Proton, 10Readers-Web-Backlog, 10MW-1.31-release-notes (WMF-deploy-2017-10-31 (1.31.0-wmf.6)): Implement Schema:Print purging strategy - https://phabricator.wikimedia.org/T175395#4057269 (10Jdlrobson) 05Open>03Resolved Thanks! [18:58:24] ottomata: hey, one more thing i forgot to ask about. Do you know anything about the mw slowparse logs? https://phabricator.wikimedia.org/T189284 [19:01:51] oo madhuvishy i don't really, i know they do still use udp2log [19:02:04] should be able to find it in site.pp etc. [19:02:16] ottomata: aah, yeah I just don't know who owns these logs [19:02:21] oh, mw folks dunno either [19:02:34] and no one seems to know (seems to be a thing ori set up) [19:05:36] (03PS4) 10Sahil505: Fixed the adblocker alert message overflow for smaller screens [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) [19:07:31] (03CR) 10Sahil505: "Yes, I somehow unknowingly deleted semantic.json file. Corrected :)" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/419977 (https://phabricator.wikimedia.org/T188208) (owner: 10Sahil505) [19:07:52] madhuvishy: here's a ticket i kinda remember, folks commeting here might know https://phabricator.wikimedia.org/T98563 [19:08:32] ottomata: thanks, lego is the only person i haven't asked, I will :) [19:16:46] 10Analytics-Tech-community-metrics: Find out if Kibana/Elasticsearch allows queries based on the results of other queries - https://phabricator.wikimedia.org/T189903#4057440 (10Aklapper) p:05Triage>03Normal [19:17:42] 10Analytics-Tech-community-metrics: Find out if Kibana/Elasticsearch allows queries based on the results of other queries - https://phabricator.wikimedia.org/T189903#4057440 (10Aklapper) [19:17:46] 10Analytics-Tech-community-metrics, 10Developer-Relations (Apr-Jun-2018): Provide Hatstall to fix syncing / updating of identities on the wikimedia.biterg.io production instance - https://phabricator.wikimedia.org/T157898#4057454 (10Aklapper) [19:17:48] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Have "Last Attracted Developers" information for Gerrit automatically updated / Integrate new demography panels in GrimoireLab product - https://phabricator.wikimedia.org/T151161#4057455 (10Aklapper) [19:18:47] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Find out if Kibana/Elasticsearch allows queries based on the results of other queries - https://phabricator.wikimedia.org/T189903#4057456 (10Aklapper) [19:18:56] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Find out if Kibana/Elasticsearch allows queries based on the results of other queries - https://phabricator.wikimedia.org/T189903#4057440 (10Aklapper) a:03Aklapper [19:39:50] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Find out if Kibana/Elasticsearch allows queries based on the results of other queries - https://phabricator.wikimedia.org/T189903#4057440 (10EBernhardson) I think the problem here is going to be the data is stored in separate indices. T... [19:45:22] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4057522 (10Ottomata) For reference, the EventLogging code that sets `userAgent.is_bot` is here: https://github.com/wikimedia/... [19:53:25] 10Analytics-Kanban: Write agreggation job for eventlogging page preview data - https://phabricator.wikimedia.org/T188310#4057537 (10Ottomata) @nuria this is ready to go for tasking, etc. [19:58:07] 10Analytics-Kanban: Write agreggation job for eventlogging page preview data - https://phabricator.wikimedia.org/T188310#4057548 (10Ottomata) https://grafana.wikimedia.org/dashboard/db/eventlogging-schema-jumbo?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-schema=VirtualPageView&from=now-24h&to=now :) [20:03:29] ya jdlrobson the units are wrong, i think it changed since the graph was switched to prometheus instead of graphite based [20:06:16] fixed [20:06:19] https://grafana-admin.wikimedia.org/dashboard/db/eventlogging-schema-jumbo?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-schema=VirtualPageView&from=1521218256477&to=1521229056477 [20:11:02] 10Analytics-Kanban: Write agreggation job for eventlogging page preview data - https://phabricator.wikimedia.org/T188310#4057592 (10Nuria) Noted, let's give the refine time to work at a scale for a few days and start on this next week. [20:17:54] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo - https://phabricator.wikimedia.org/T189464#4057613 (10Ottomata) a:03Ottomata [20:23:36] !log bouncing main -> jumbo mirror makers to apply change-prop topic blacklist [20:23:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:30:26] 10Analytics, 10Community-Tech: Data missing in page creation datasets - https://phabricator.wikimedia.org/T185019#4057643 (10Ottomata) [20:30:28] 10Analytics-Kanban: Investigate why data was missing from mediawiki events around January 3rd - https://phabricator.wikimedia.org/T185229#4057639 (10Ottomata) 05Open>03Resolved I think likely what happened is mirror maker was late. I'm going to close this task for now, as the parent was closed. We need a... [20:44:34] Gone for tonight a-team [20:44:50] byeeee, have a nice weekend! [22:13:28] 10Analytics-Kanban, 10MW-1.31-release-notes (WMF-deploy-2018-02-27 (1.31.0-wmf.23)), 10Patch-For-Review: Record and aggregate page previews - https://phabricator.wikimedia.org/T186728#4057779 (10Jdlrobson) For those following along - T188310 tracks the aggregation job and my team should work to carefully rol... [22:49:33] Nettrom: SUPER nice Job on ACTRIAL research [22:49:57] @nuria_ : thanks so much :)