[07:13:27] <wikibugs>	 Analytics, RESTBase, Services, User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2242082 (mobrovac) I like the general direction, @JAllemandou . But the first one should really go under `/{domain}/api/rest_v...
[07:23:03] <wikibugs>	 Analytics-Kanban, Patch-For-Review: Out of service banner in dashiki - https://phabricator.wikimedia.org/T133736#2242100 (elukey)
[07:23:05] <wikibugs>	 Analytics-Kanban, Operations, Patch-For-Review: Upgrade stat1001 to Debian Jessie - https://phabricator.wikimedia.org/T76348#2242099 (elukey)
[07:31:07] <elukey>	 o/
[07:31:59] <elukey>	 mmmmm dashiki dashboards also down?
[07:32:10] <elukey>	 (during stat1001 maintenance)
[07:32:36] <elukey>	 I am wondering if it would be the case of having proper redundancy
[07:32:49] <elukey>	 and not only one host for everything
[08:13:04] <joal>	 elukey: I guess if proper redundancy was setup, the systems would support one of the machines being down (for whatever reason)
[08:14:12] <elukey>	 yepppp
[08:14:38] <elukey>	 could be tricky though, but from what I can read stat1001 seems really important for us
[08:19:25] <wikibugs>	 Analytics, RESTBase, Services, User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2242183 (JAllemandou) @mobrovac : Interesting. In my opinion the main concern of endpoint conceptual coherency applies here to...
[08:20:20] <joal>	 elukey: stat1001 is the holly grail of the a-team
[08:20:36] <joal>	 The a-team dies for stat1001, without even a question ! ;)
[08:21:22] <joal>	 Hey mobrovac, would you give me a minute to discuss the numeric format issue (not solve, discuss first :)
[08:21:25] <elukey>	 joal: :)
[08:21:45] <mobrovac>	 joal: go go go :)
[08:22:00] <mobrovac>	 you have 60 seconds, start
[08:22:01] <mobrovac>	 hahaha
[08:22:29] <mobrovac>	 i always wondered if people actually say that in reality or is just a movies thing
[08:22:36] <joal>	 mobrovac: To put it simply, when using long number formats, cassandra returns a string and sqlite returns a number --> Which of the two do you consider correct, and therefore should be expected in the code ?
[08:22:40] <joal>	 mobrovac: DONE @!
[08:22:44] <joal>	 Muhahaha
[08:22:44] <mobrovac>	 lol
[08:22:46] <joal>	 :)
[08:23:02] <mobrovac>	 ah yes! longs!
[08:23:21] <mobrovac>	 we had problems with them because cassandra and nodejs encode/decode them differently
[08:23:31] <mobrovac>	 so we opted for saving them as strings in cassandra
[08:24:15] <mobrovac>	 this is basically to prevent overflow issues
[08:25:48] <joal>	 ok mobrovac
[08:26:02] <mobrovac>	 mystery solved :)
[08:26:10] <mobrovac>	 thnx for reminding me joal of this "feature"
[08:26:10] <joal>	 mobrovac: So the epected behavior is to decode string and reencode them as number
[08:26:18] <joal>	 mo
[08:26:36] <joal>	 mobrovac: Cause when using sqlite, numbers are returned ;)
[08:27:54] <joal>	 Actually mobrovac: looking into CQL: the long defined numbers are defined as bigint in the schema
[08:28:35] <joal>	 mobrovac: The encoding change happens in the restbase-cassandra module?
[08:29:05] <mobrovac>	 yes joal
[08:29:15] <mobrovac>	 i think we should change the sqlite mod to be in line with cassandra
[08:29:52] <mobrovac>	 joal: the actual problem is that cardinality of bigint in cass is bigger than the one of JS' Number
[08:29:58] <joal>	 mobrovac: ok, that makes sense then (I don't want to question why an encoding change from long to string in node ;)
[08:30:04] <mobrovac>	 so there are edge cases where you might get stuff like -1000
[08:30:25] <joal>	 Ok, didn't want to question, got an answer, you're awesome mobrovac :)
[08:30:38] <joal>	 makes sense
[08:31:04] <joal>	 I'll modify the aqs code and tests to assume sqlite module returns a string as well
[08:31:52] <joal>	 mobrovac: how do you decode longs in node to have a proper number if cassandra return a too big stuff ? Byte level decoding?
[08:34:05] <mobrovac>	 we do some binary hacks, but tbh i'm not sure we're still using them
[08:34:09] <mobrovac>	 i'd need to check joal
[08:34:31] <joal>	 mobrovac: I was just curious :)
[08:34:45] <mobrovac>	 :)
[08:35:00] <joal>	 mobrovac: As said, problem solved, I'll assume sqlite returns a string and we're good :)
[08:35:15] <joal>	 mobrovac: I let you create a task to correct the sqlite module bug?
[08:35:24] <mobrovac>	 ok, we'll try to get a new version soon that actually does that
[08:35:26] <mobrovac>	 yup joal
[08:35:31] <joal>	 awesome :)
[08:35:35] <joal>	 Thanks mobrovac !
[08:35:38] <mobrovac>	 np
[08:35:45] <mobrovac>	 thank you!
[08:35:54] <joal>	 ;)
[08:47:16] <wikibugs>	 Analytics, Pageviews-API, I18n: [[Wikimedia:Pageviews-num-languages/en]] needs PLURAL - https://phabricator.wikimedia.org/T133766#2242217 (Liuxinyu970226)
[08:54:23] <grrrit-wm>	 (PS3) Joal: Correct unique devices endpoint value format [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 (https://phabricator.wikimedia.org/T133527)
[08:56:46] <grrrit-wm>	 (CR) Joal: "Discussed with @mobrovac: For the moment decoding unit test is still not valid due to some encoding/decoding inconsistency in sqlite modul" [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 (https://phabricator.wikimedia.org/T133527) (owner: Joal)
[08:58:20] <mobrovac>	 joal: fyi, if you have a failing test that you don't want to execute, but want to keep around, you can use it.skip() instead of it() in the tests
[08:58:37] <joal>	 mobrovac: Yay !
[08:58:50] <joal>	 mobrovac: actually, test doesn't fail but doesn't test anything ;)
[08:59:06] <mobrovac>	 :)
[08:59:11] <mobrovac>	 TDD done right!
[08:59:38] <joal>	 YAY mobrovac :)
[09:03:01] <joal>	 elukey: meeting ?
[09:03:05] <elukey>	 joal: yep
[09:41:48] <joal>	 elukey: Just realized one thing ...
[09:41:56] <joal>	 elukey: batcave again for a minute?
[09:41:56] * elukey is scared
[09:42:07] <elukey>	 sure!
[09:42:09] <joal>	 no no no, actyually make it easier :)
[09:47:40] <grrrit-wm>	 (CR) Joal: [C: 1] "Looks good, thanks !" [analytics/aqs/deploy] - https://gerrit.wikimedia.org/r/285535 (https://phabricator.wikimedia.org/T132267) (owner: Alex Monk)
[10:25:19] <wikibugs>	 Analytics, DBA: Set up auto-purging after 90 days {tick} - https://phabricator.wikimedia.org/T108850#1532166 (jcrespo) a:jcrespo
[10:32:07] * elukey lunch!
[10:42:48] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2242437 (jcrespo) I've checked the structure of the tables, and it was applied to all tables on the CSV, but 2, which no longer exist:  ```  MobileWebClickTracking_5830671: ERROR 1146 (42S02) at li...
[11:00:10] <mforns>	 joal, hi yt?
[11:01:02] <grrrit-wm>	 (CR) Mforns: [C: 1] "LGTM!" [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 (https://phabricator.wikimedia.org/T133527) (owner: Joal)
[11:03:05] <joal>	 Hi mforns :)
[11:03:25] <joal>	 wasup mforns ?
[11:03:45] <mforns>	 hey joal, I reviewed the aqs change and looks good, I didn't +2 because I don't know if this will automatically deploy the change?
[11:03:58] <joal>	 mforns: it will not :)
[11:04:03] <mforns>	 ok
[11:04:12] <joal>	 aqs deploy is manual, with scap3, but manual
[11:05:33] <mforns>	 I see
[11:12:03] <grrrit-wm>	 (CR) Mforns: [C: 2 V: 2] "LGTM!" [analytics/aqs] - https://gerrit.wikimedia.org/r/285180 (https://phabricator.wikimedia.org/T133527) (owner: Joal)
[11:12:23] <elukey>	 mforns: helloooo
[11:13:02] <elukey>	 I saw the backlog between you and nuria_ and I was wondering if I need to do something for the dashiki stuff
[11:13:42] <joal>	 Thanks mforns !
[11:13:49] <mforns>	 elukey, hi!
[11:13:55] <mforns>	 np joal
[11:14:14] <mforns>	 elukey, I don't think so
[11:14:41] <mforns>	 the only changes needed will be in dashiki and the config wiki pages
[11:15:31] <elukey>	 mforns: super, are you going to change them? I am trying to write down the things to do before starting the re-image on monday :)
[11:18:00] <wikibugs>	 Analytics-Kanban: Add unique devices to vital signs - https://phabricator.wikimedia.org/T133771#2242483 (mforns)
[11:21:25] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2242503 (jcrespo) The following trigger was created on `MobileWebWatchlistClickTracking_10720361`:   ``` SET SESSION sql_log_bin = 0; CREATE TRIGGER MobileWebWatchlistClickTracking_10720361_editCou...
[11:23:05] <mforns>	 elukey, nuria_ said today we would be able to code review the Dashiki changes, and probably deploy that. At the moment of the downtime, we'll just need to edit the wiki config page, and the banner will show
[11:23:51] <elukey>	 niceeeee
[11:24:10] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2242509 (jcrespo) It seems that table is no longer active, if you have an example of another table to test, feel free.
[11:24:12] <elukey>	 if you could comment what to do in the phab task before Monday I'll be super happy :)
[11:24:26] <mforns>	 elukey, the edition of the config wiki, I can do it if I'm online at the time, oh ok
[11:24:45] <mforns>	 I can put 2 lines of instructions on how to make the banner show and disappear
[11:25:06] <mforns>	 elukey, will do that today after reviewing the Dashiki changes
[11:26:17] <mforns>	 team, have to leave now for a while, cya
[12:05:21] <wikibugs>	 Analytics-EventLogging, TimedMediaHandler, Wikimedia-Video: Record and report metrics for audio and video playback - https://phabricator.wikimedia.org/T108522#2242595 (TheDJ) Some of the archive.org stats: https://archive.org/~tracey/mrtg/  {F3936348}
[12:26:59] <elukey>	 a-team: need to go afk for maximum 1 hour, ttl!
[12:38:19] <wikibugs>	 Analytics, RESTBase, Services, User-mobrovac: configure RESTBase pageview proxy to Analytics' cluster on wiki-specific domains - https://phabricator.wikimedia.org/T119094#2242646 (mobrovac) I think the misunderstanding here comes form the POV: you look it from the //I'm a pageview API user// and...
[13:53:39] <elukey>	 !log restarted kafka on kafka1018.eqiad.wmnet for Java upgrades
[13:53:41] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[13:55:22] * elukey waits for event logging
[13:55:56] <elukey>	 EL seems fine
[13:55:57] <elukey>	 woa
[13:57:32] <elukey>	 now I need to restart all the other nodes, any bet if EL will complain? :P
[13:57:54] * elukey thinks that EL doesn't simply like him
[13:58:10] <icinga-wm>	 PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: forwarder/legacy-zmq
[13:58:19] <elukey>	 buuuuuuuu
[13:59:36] <elukey>	 socket.error: [Errno 111] Connection refused
[14:01:41] <elukey>	 !log restarted Event Logging on eventlogging1001
[14:01:42] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[14:04:05] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning.
[14:10:35] <wikibugs>	 Analytics-Kanban: Event Logging doesn't handle kafka nodes restart cleanly - https://phabricator.wikimedia.org/T133779#2242892 (elukey)
[14:31:25] <Krenair>	 joal, hey
[14:31:31] <Krenair>	 what do I need to get https://gerrit.wikimedia.org/r/#/c/285535/ merged?
[14:46:06] <elukey>	 mmmm I can't find eventlogging's logs anymore for EventBus in /var/log, and I can't find anything useful in /srv
[15:12:12] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2243055 (Nuria) @jcrespo; Thank you, so all other tables listed on spreadsheet but this one have been updated with edit  count, then. Correct?
[15:16:30] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Use MySQL as Hue data backend store - https://phabricator.wikimedia.org/T127990#2243059 (elukey) @Ottomata: we have a choice to make:  1) don't use TLS and complete this task, enabling it maybe in the future.  2) use TLS and configure it properly...
[15:22:03] <elukey>	 mforns: can you give me some links of reports not working when stat1001 will be down?
[15:22:05] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2243065 (Nuria) @jcrespo: nevermind my question i see it now where it pertains:  mysql:research@analytics-store.eqiad.wmnet [log]> desc MobileWebDiffClickTracking_10720373; +---------------------+-...
[15:22:09] <elukey>	 just to be sure
[15:23:10] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2243066 (Nuria) What would be the best way to add an editcount column to new schemas going forward? Should we have an audit once every 6 months to find tables that might need it?
[15:28:29] <nuria_>	 elukey: anything on stats.wikimedia.org will no longer work
[15:29:27] <elukey>	 nuria_: okok I thought also http://reportcard.wmflabs.org/
[15:30:00] <nuria_>	 elukey: that page wil show, it will have no data though, but we no longer support it so it will not have a banner either
[15:30:34] <nuria_>	 elukey: the new dashboards on dashiki, like: https://vital-signs.wmflabs.org/#projects=ruwiki/metrics=Pageviews should have a banner
[15:31:46] <elukey>	 ahhhh okok thanks! Sorry I still need to memorize all the webpages :)
[15:31:49] <elukey>	 https://grafana.wikimedia.org/dashboard/db/kafka?panelId=29&fullscreen
[15:31:56] <elukey>	 one partition not in sync grrr
[15:33:50] <elukey>	 Topic: webrequest_uploadPartition: 6Leader: 14Replicas: 14,20,22Isr: 14,20
[15:37:57] <nuria_>	 elukey: that is why we need the new analytics.wikimedia domain to have all dashboards organized there
[15:38:24] <elukey>	 nuria_: another thing that I am wondering is if we need proper replication for stat1001
[15:38:44] <elukey>	 because it seems really really important for us
[15:39:07] <elukey>	 any problem to the host that is not trivial can impair our tools
[15:39:10] <nuria_>	 elukey: we can talks about that with otto as analytics machines are big massive machines to do computations that are also used to serve data
[15:39:32] <elukey>	 yep!
[15:39:41] <nuria_>	 elukey: so can we have a cluster of 1001-like machines? probably not, but that doesn't mean we have to serve all our data from there
[15:39:47] <nuria_>	 elukey: but this is a question for later
[15:45:20] <elukey>	 nuria_: two hosts behind lvs/pybal could do the trick but data should be replicated/rsynched/whatever everywhere..
[15:45:48] <nuria_>	 elukey: right, but we do not have hardware for a 1001-like machine that i know of
[15:46:09] <icinga-wm>	 PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: processor/client-side-04 processor/client-side-01 forwarder/legacy-zmq
[15:46:18] <elukey>	 I was expecting you
[15:47:01] <nuria_>	 jaja
[15:47:10] <elukey>	 !log restarted event logging on eventlogging1001
[15:47:12] <analytics-logbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log, Master
[15:47:52] <elukey>	 this is a big problem, EL is again really sensible to kafka restarts :(
[15:48:19] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning.
[15:50:19] <elukey>	 either we check pykafka again or maybe add some robustness to /usr/lib/python2.7/dist-packages/pykafka/broker.py
[15:52:00] <elukey>	 ah snap it is indeed in pykafka
[15:52:38] <nuria_>	 elukey: would you file a ticket for the issue with restarts? Thank youuu
[15:53:45] <elukey>	 nuria_: already done :
[15:53:46] <elukey>	 :)
[15:53:52] <nuria_>	 thanks
[15:57:51] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Self merging to provide working code after next merge." [analytics/wikihadoop] - https://gerrit.wikimedia.org/r/251311 (https://phabricator.wikimedia.org/T114359) (owner: Joal)
[15:58:31] <grrrit-wm>	 (CR) Joal: [C: 2 V: 2] "Self merging to provide working xmljson job in mapreduce" [analytics/wikihadoop] - https://gerrit.wikimedia.org/r/285203 (owner: Joal)
[15:59:38] <wikibugs>	 Analytics: Provision new SSD-able machines on AQS - https://phabricator.wikimedia.org/T132938#2243178 (RobH)
[15:59:40] <wikibugs>	 Analytics, Analytics-Cluster, Operations, hardware-requests: Hadoop Node expansion for end of FY - https://phabricator.wikimedia.org/T124951#2243179 (RobH)
[15:59:42] <wikibugs>	 Analytics-Kanban, RESTBase-Cassandra: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#2243180 (RobH)
[15:59:46] <wikibugs>	 Analytics, Operations, hardware-requests, Patch-For-Review: eqiad: (3) AQS replacement nodes - https://phabricator.wikimedia.org/T124947#2243176 (RobH) Open>Resolved They've arrived onsite, and are in the queue for chris to rack.  I'm marking this as resolved by the purchase task T132067.
[16:00:13] <nuria_>	 a-team: standdduppp
[16:00:23] <nuria_>	 madhuvishy, joal: hola!
[16:01:28] <bmansurov>	 lzia: i'm online
[16:01:45] <lzia>	 hey bmansurov. I was going to message you, sorry.
[16:01:58] <lzia>	 I'm going to move this event to next week, to finish the self-one first, and then do this.
[16:01:58] <bmansurov>	 lzia: np if you can't make it
[16:01:59] <lzia>	 is this cool?
[16:02:08] <lzia>	 And I'll email you when I'm done so you know it.
[16:02:13] <bmansurov>	 lzia: the deadline is the 29th though
[16:02:22] <lzia>	 no, that's the deadline for nominations.
[16:02:28] <lzia>	 the actual submissions are at May 6
[16:02:31] <lzia>	 on*
[16:02:33] <bmansurov>	 lzia: oh cool then
[16:02:53] <bmansurov>	 lzia: i'll be on offsite next week, but we can work something out
[16:02:54] <lzia>	 Everything is due on the 6th. They are encouraging the self-reviews to finish this week.
[16:03:04] <bmansurov>	 lzia: ok, sounds good
[16:03:11] <lzia>	 we don't need to meet, I'll send you an email just fyi once I submit it, bmansurov
[16:03:23] <bmansurov>	 lzia: ok that's even better
[16:03:30] <bmansurov>	 lzia: thank you so much
[16:03:35] <lzia>	 sorry for not updating the event bmansurov.
[16:03:40] <lzia>	 np. :)
[16:03:42] <bmansurov>	 lzia: np ;)
[16:05:57] <wikibugs>	 Analytics-Kanban, RESTBase-Cassandra: Better response times on AQS (Pageview API mostly) {melc} - https://phabricator.wikimedia.org/T124314#2243211 (RobH)
[16:05:59] <wikibugs>	 Analytics: Visualize unique devices data in dashiki {bear} - https://phabricator.wikimedia.org/T122533#2243213 (Nuria) a:mforns
[16:06:48] <wikibugs>	 Analytics-Kanban: Visualize unique devices data in dashiki {bear} - https://phabricator.wikimedia.org/T122533#1906757 (Nuria)
[16:07:37] <joal>	 halfak: One request on altiscale: can you update the /wmf/jars/wikihadoop-0.2.jar to /tmp/wikihadoop/wikihadoop-0.2.jar ?
[16:07:39] <wikibugs>	 Analytics-Kanban: Add unique devices to vital signs - https://phabricator.wikimedia.org/T133771#2243221 (mforns)
[16:07:44] <joal>	 Like that, default will work !
[16:08:18] <wikibugs>	 Analytics, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2243229 (RobH)
[16:09:04] <wikibugs>	 Analytics, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2243191 (RobH)
[16:09:53] <elukey>	 wwoooaaa
[16:10:00] <elukey>	 joal: --^
[16:10:14] <joal>	 YAYYYYYYY !
[16:10:32] <joal>	 elukey: perfectly one time, now that we have a plan ;)
[16:11:01] <joal>	 Krenair: Sorry, many meetings this afternoon
[16:11:19] <joal>	 Krenair: I'd like nuria or andrew to +1 as well, and then can merge :)
[16:11:28] <Krenair>	 joal, okay, thanks
[16:12:14] <joal>	 ottomata, added you to https://gerrit.wikimedia.org/r/#/c/285535/ (for Krenair)
[16:22:35] <wikibugs>	 Analytics-EventLogging, MobileFrontend, Reading-Web, Easy, and 2 others: Should be possible to override sampling in EventLogging schemas for development purpose - https://phabricator.wikimedia.org/T125122#2243416 (MBinder_WMF)
[16:26:27] <madhuvishy>	 nuria_: I think I'll skip grooming and get to office - have to talk to amanda, and more meetings in the evening. Okay with you?
[16:27:13] <nuria_>	 madhuvishy: sure, let's please not work on wikimetrics besides assessing anddocumenting whether there is a problem
[16:27:33] <madhuvishy>	 nuria_: yup
[16:29:42] <wikibugs>	 Analytics, Operations, hardware-requests, Patch-For-Review: eqiad: (3) AQS replacement nodes - https://phabricator.wikimedia.org/T124947#2243639 (Ottomata) @cmjohnson if you could prioritize this one a little, we'd appreciate it.  We've been waiting for a while and the current OOW nodes that are...
[16:34:11] <bd808>	 The instructions on https://wikitech.wikimedia.org/wiki/Analytics/EventLogging/Data_representations are not completely clear to me on how to access the mysql version of EL data. Do I need to have access to stat1003 to get to that or is there a way to query it from stat1002?
[16:34:33] <bd808>	 I can get at data via hive queries but it seems like mysql would be easier for the things I'm looking for today
[16:35:51] <wikibugs>	 Analytics, Operations, hardware-requests, Patch-For-Review: eqiad: (3) AQS replacement nodes - https://phabricator.wikimedia.org/T124947#2243827 (Ottomata) We can call these aqs100[456].  If you can just get these to DNS and ready for install, we will handle the actual partman layout and install....
[16:36:03] <wikibugs>	 Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2243191 (Nuria)
[16:36:05] <wikibugs>	 Analytics: Event Logging doesn't handle kafka nodes restart cleanly - https://phabricator.wikimedia.org/T133779#2243842 (elukey)
[16:38:56] <wikibugs>	 Analytics, Pageviews-API: Provide weekly top pageviews stats - https://phabricator.wikimedia.org/T133575#2236232 (Nuria) A new job to add into cassandra loading. More than that a big change on how to present data vi api. Thus far we only present data daily and monthly.
[16:41:24] <wikibugs>	 Analytics: EventLogging suffers for Kafka broker restarts - https://phabricator.wikimedia.org/T132922#2243924 (Nuria)
[16:41:26] <wikibugs>	 Analytics: Event Logging doesn't handle kafka nodes restart cleanly - https://phabricator.wikimedia.org/T133779#2243925 (Nuria)
[16:43:38] <wikibugs>	 Analytics-Cluster, Operations, ops-eqiad: Analytics hosts showed high temperature alarms - https://phabricator.wikimedia.org/T132256#2192798 (Nuria)
[16:51:41] <wikibugs>	 Analytics: Pageview API alarms - https://phabricator.wikimedia.org/T132456#2243984 (Nuria) The current alarms provided by REST base are insufficient
[16:55:37] <wikibugs>	 Analytics, Operations, Traffic: cronspam from cpXXXX hosts related to varnishkafka non existent processes - https://phabricator.wikimedia.org/T132346#2243999 (Nuria) Open>Resolved
[17:00:52] <mforns>	 elukey, do you want a comprehensive list of all the dashboards?
[17:00:57] <mforns>	 or just a few of them?
[17:02:48] <elukey>	 mforns: I'll use https://stats.wikimedia.org/EN/SummarySIMPLE.htm as example and that's it, it should be enough.. what do you think?
[17:03:00] <joal>	 elukey: looks like we're going to play with 2 instances instead of 4
[17:03:19] <elukey>	 joal: it'll be fun anyway :P
[17:03:45] <mforns>	 elukey, ?
[17:04:26] <elukey>	 mforns: did you see my post above?
[17:04:41] <mforns>	 elukey, oh sorry
[17:05:05] <joal>	 elukey: very true 1
[17:05:16] <joal>	 halfak: have you seen my previous message?
[17:05:19] <mforns>	 elukey, I thought you needed the Dashiki/Limn dashboards that get data files from datasets.wmflabs.org?
[17:06:00] <elukey>	 mmmmmmmmm
[17:06:15] <elukey>	 all right mfors give me some links :P
[17:06:38] <elukey>	 like a couple of them
[17:06:46] <mforns>	 elukey, ok
[17:07:09] <joal>	 ottomata: you were not online when I tried to ring you  :)
[17:07:20] <ottomata>	 i just realized!
[17:07:23] <joal>	 ottomata: do you mind having a look at https://gerrit.wikimedia.org/r/#/c/285535/ for Krenair ?
[17:07:26] <ottomata>	 that i wasn't signed into irc\
[17:07:34] <ottomata>	 looking
[17:07:43] <joal>	 Thanks :)
[17:08:04] <mforns>	 elukey, vital-signs.wmflabs.org, browser-reports.wmflabs.org, mobile-reportcard.wmflabs.org, ee-dashboard.wmflabs.org/dashboards/enwiki-features, edit-analysis.wmflabs.org/compare
[17:08:20] <ottomata>	 oh iiinteresting!
[17:08:29] <ottomata>	 i didn't know scap could do the per env overrides like that
[17:08:40] <ottomata>	 i thought you had to give it  on the cli as an opt
[17:09:15] <elukey>	 mforns: thanks!
[17:09:22] <grrrit-wm>	 (CR) Ottomata: [C: 1] Fix up deployment-prep scap config [analytics/aqs/deploy] - https://gerrit.wikimedia.org/r/285535 (https://phabricator.wikimedia.org/T132267) (owner: Alex Monk)
[17:09:43] <mforns>	 elukey, np
[17:10:00] <wikibugs>	 Analytics, Hovercards, Reading-Web, Reading-Web-Sprint-71-Matisse-Monet-Kandinsky-and-the-Departing-Painters: Verify X-Analytics: preview=1 in stable - https://phabricator.wikimedia.org/T133067#2244028 (MBinder_WMF)
[17:12:17] <joal>	 Krenair: Quick question for you
[17:12:24] <Krenair>	 ok
[17:12:45] <joal>	 Krenair: ottomata has reviewed your patch, I'll merge and actuallytry it tomorrow since I want to deploy :)
[17:13:09] <Krenair>	 ok
[17:13:19] <joal>	 I wonder how to deploy on prod and not in deployment, or vice-versa
[17:13:28] <elukey>	 ottomata: I didn't find request logs on EventBus today
[17:13:39] <elukey>	 did something changed recently?
[17:13:45] <elukey>	 with the scap deployment, etc..
[17:13:57] <joal>	 Krenair: Does that happen automatically given which git server is used?
[17:14:00] <ottomata>	 elukey:  ja i quieted request logs there, getting too noisy
[17:14:02] <elukey>	 I mean, I can't see anymore all the requests
[17:14:05] <elukey>	 ahhhhhh
[17:14:17] <elukey>	 we'd need to update https://wikitech.wikimedia.org/wiki/EventBus/Administration
[17:14:28] <Krenair>	 joal, prod and beta can't touch each other
[17:14:40] <ottomata>	 ahhhh ok!
[17:14:47] <Krenair>	 if you deploy in prod it'll go to prod hosts, if you deploy in beta it'll go to beta hosts
[17:14:53] <ottomata>	 will do
[17:14:55] <elukey>	 going offline a-team!
[17:15:07] <mforns>	 bye elukey !
[17:15:08] <joal>	 Automagically using git servers provided in config Krenair, right ?
[17:15:14] <joal>	 Bye elukey ! Tomorrow :)
[17:15:24] <joal>	 elukey: Thanks for the good cassandra time :)
[17:15:25] <elukey>	 ottomata: I have posted comments for hue and conf200X hosts, if you have time let me know your thoughts :)
[17:15:31] <Krenair>	 sort of, yes
[17:15:38] <joal>	 ok
[17:15:40] <ottomata>	 elukey:  i am reviewing gerrit stuff now ;:)
[17:15:50] <elukey>	 joal: it will be a nice journey joal! A lot of swearing and eventually beers
[17:15:51] <joal>	 Krenair: So no need for me to bother too much
[17:15:53] <elukey>	 :P
[17:16:06] <Krenair>	 it's already in beta
[17:16:11] <joal>	 indeed elukey, proper journeys are never easy ;)
[17:16:21] <joal>	 But how good is the beer after !
[17:16:23] <Krenair>	 just needs to go into the repo properly at some point and therefore be in prd as well
[17:16:24] <Krenair>	 prod*
[17:16:51] <joal>	 hm ... Currently I deploy from tin, therefore in prod
[17:16:56] <joal>	 Krenair: --^
[17:17:04] <Krenair>	 yes
[17:17:13] <joal>	 Krenair: From where shall I deploy if I want to deploy in beta first ?
[17:17:28] <Krenair>	 deployment-tin.deployment-prep.eqiad.wmflabs
[17:17:36] <joal>	 Krenair: awesome :)
[17:17:45] <joal>	 Krenair: Will test tomorrow morning
[17:18:51] <joal>	 Thanks a lot Krenair :)
[17:19:09] <Krenair>	 no problem
[17:19:12] <joal>	 a-team, logging off as well
[17:19:17] <joal>	 See you tomorrow !
[17:19:21] <mforns>	 bye joal !
[17:20:16] <ottomata>	 mobrovac: hey
[17:20:21] <ottomata>	 can we change this?
[17:20:21] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/275772/7/manifests/role/changeprop.pp
[17:20:32] <ottomata>	 to use kafka_config rather than hardcoding zookeeper_url in hiera?
[17:21:00] <mobrovac>	 i thought i changed that
[17:21:02] <mobrovac>	 hm
[17:21:19] <mobrovac>	 i will if i haven't yet done so
[17:21:42] <ottomata>	 danke, we shoudl do that before elukey sets up the new zookeeper hosts and main kafka cluster in codfw
[17:21:50] <ottomata>	 and remove zookeeper_url from hiera
[17:21:55] <ottomata>	 oh
[17:21:58] <ottomata>	 maybe you did mobrovac
[17:22:03] <ottomata>	 looking
[17:22:06] <ottomata>	 i was looking at blame
[17:22:10] <ottomata>	 for the zookeeper_hosts
[17:22:12] <ottomata>	 in hiera
[17:22:22] <ottomata>	 mobrovac:  cool, yeah you already did
[17:22:22] <ottomata>	 sorry
[17:25:45] <ottomata>	 elukey:  i dont' see the hue review on my gerrit incoming list
[17:25:50] <ottomata>	 have you addeed me as reviewer?
[17:25:55] <ottomata>	 oh, or you mean on phab?
[17:26:14] <ottomata>	 on phab duh
[17:26:47] <ottomata>	 elukey:  why does it need to be encrypted with tls?
[17:30:32] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Patch-For-Review: Use MySQL as Hue data backend store - https://phabricator.wikimedia.org/T127990#2244157 (Ottomata) Hm, I'd choose option 1.  We don't (yet) encrypt other types of traffic within the analytics cluster, and the Hue queries will be no more sensitive...
[17:33:44] <grrrit-wm>	 (PS5) Nuria: Add out of service banner to dashiki [analytics/dashiki] - https://gerrit.wikimedia.org/r/285255 (https://phabricator.wikimedia.org/T133736)
[17:34:03] <nuria_>	 mforns: dashiki banner is ready to be CR-ed and tested
[17:34:15] <mforns>	 nuria_, will do
[17:39:58] <nuria_>	 mforns: you can change config here: https://meta.wikimedia.org/wiki/Dashiki:OutOfService
[17:42:52] <mforns>	 nuria_, saw that in the commit message, thanks!
[17:50:32] <grrrit-wm>	 (CR) Ottomata: "Hm, cool! I was hesitant about putting stuff after the 'coord', like 'coord-....', but I think I see the intent: If there is more specif" [analytics/refinery] - https://gerrit.wikimedia.org/r/285400 (owner: Joal)
[17:53:55] <halfak>	 joal, just saw your messages.  Moving the file now
[17:54:13] <halfak>	 Sorry.  Meetings and then a deployment to deal with today.
[17:55:07] <halfak>	 {{done}}
[17:59:54] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2244265 (jcrespo) This is the full list of triggers I am going to create on the master. Only before insert, assuming no updates take place, ever: ``` $ ./eventlogging_triggers.sh   SET SESSION sql_...
[18:10:15] <grrrit-wm>	 (CR) Mforns: [C: -1] "LGTM overall, but there are a couple details (see comments). There is one thing that I could not comment inline: We should add: 'component" (5 comments) [analytics/dashiki] - https://gerrit.wikimedia.org/r/285255 (https://phabricator.wikimedia.org/T133736) (owner: Nuria)
[18:11:12] <mforns>	 nuria_, I did the CR, there are a couple details, when I come back from the gym I can look at it again, if you want
[18:18:57] <wikibugs>	 Analytics, DBA: Set up bucketization of editCount fields {tick} - https://phabricator.wikimedia.org/T108856#2244410 (jcrespo) These are the triggers created, as you can see 13, 2 less than the list proposed due to 2 nonexistent tables: ``` MariaDB  db1046.eqiad.wmnet log > SELECT trigger_name, EVENT_OBJE...
[18:56:48] <joal>	 halfak: no problemo, thanks for the update :)
[19:00:32] <wikibugs>	 Analytics, ArchCom-RfC, Discovery, EventBus, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#2244571 (csteipp)
[19:03:52] <grrrit-wm>	 (CR) Joal: "@Ottomata: Actually I changed _ and - to match the following pattern: /dir1/dir2_option2/bundle.xml is named dir1-dir2_option2-bundle" [analytics/refinery] - https://gerrit.wikimedia.org/r/285400 (owner: Joal)
[19:05:41] <grrrit-wm>	 (PS2) Joal: Normalize oozie job names (bundles, coords, wfs) [analytics/refinery] - https://gerrit.wikimedia.org/r/285400
[19:08:33] <wikibugs>	 Analytics-Kanban: Standardise naming in oozie jobs (particularly for top level ones) - https://phabricator.wikimedia.org/T130732#2244610 (JAllemandou) a:JAllemandou
[19:09:05] <grrrit-wm>	 (PS3) Joal: Normalize oozie job names (bundles, coords, wfs) [analytics/refinery] - https://gerrit.wikimedia.org/r/285400 (https://phabricator.wikimedia.org/T130732)
[19:09:58] <wikibugs>	 Analytics-Kanban: Make webrequest load and refine jobs a single bundle - https://phabricator.wikimedia.org/T130731#2244616 (JAllemandou) a:JAllemandou
[19:12:17] <wikibugs>	 Analytics, ArchCom-RfC, Discovery, EventBus, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#2244625 (mobrovac)
[19:12:45] <wikibugs>	 Analytics, Discovery, EventBus, MediaWiki-General-or-Unknown, and 6 others: Reliable publish / subscribe event bus - https://phabricator.wikimedia.org/T84923#2244628 (mobrovac)
[19:12:51] <wikibugs>	 Analytics, ArchCom-RfC, Discovery, EventBus, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#1711519 (mobrovac) Open>Resolved And we're done here!
[19:31:30] <nuria_>	 mforns_gym: fixing and resubmitting
[19:31:57] <nuria_>	 madhuvishy: hola! did you had opportunity to talk to amanda?
[19:32:13] <madhuvishy>	 nuria_: not yet, we are chatting at 2pm
[19:32:20] <nuria_>	 madhuvishy: k
[19:42:38] <grrrit-wm>	 (PS6) Nuria: Add out of service banner to dashiki [analytics/dashiki] - https://gerrit.wikimedia.org/r/285255 (https://phabricator.wikimedia.org/T133736)
[20:57:05] <wikibugs>	 Analytics, ArchCom-RfC, Discovery, EventBus, and 7 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#2244925 (Ottomata) YEEHAW
[21:42:28] <wikibugs>	 Analytics-Kanban, Operations, ops-eqiad: rack/setup/deploy aqs100[456] - https://phabricator.wikimedia.org/T133785#2245119 (Ottomata) They should be in separate racks from each other, but these are replacements for aqs100[123], so it doesn't matter if they are in the same racks as those.  If yall can...