[00:17:21] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Merge ~1350 duplicated Phab accounts in the Bitergia DB - https://phabricator.wikimedia.org/T179745#3870154 (10Aklapper) The query above shows zero results, but `SELECT p.uuid,mw.uuid,p.name,mw.name,p.username,mw.username FROM identities... [00:53:54] 10Analytics, 10Operations, 10Research, 10Traffic, and 6 others: Referrer policy for browsers which only support the old spec - https://phabricator.wikimedia.org/T180921#3870196 (10Nuria) >Flipping these Edges/Safaris to origin is going to deny us information on internal referrers that we currently get from... [08:59:20] hello people [08:59:29] morning elukey [08:59:43] as FYI I am stopping the mysql consumer on eventlog1001 [09:02:35] ok elukey [09:04:22] so just to recap [09:04:41] stopped mysql-consumer and eventbus-mysql consumer on eventlog1001 (and disabled puppet) [09:04:53] stopped eventlogging_sync on db1108 and disabled puppet [09:06:21] and removed the /var/run/eventlogging_cleaner file on db1107 to prevent eventlogging_cleaner to run [09:08:59] elukey: next step is to apply UPDATE script? [09:10:32] yep exactly [09:10:49] Yay, I'm not completely misunderstanding :) [09:17:36] started! [09:19:22] it should take a couple of days more or less [09:19:45] man ... This is long [09:20:22] elukey: I have updated the example metric for the alert in the dashboard, if you could have a look [09:20:28] sure! [09:20:50] elukey: It gives me weird results when ran from grafana, but I assume it would have correct ones from puppet ? [09:26:55] joal: what do you mean with weird results? [09:27:10] elukey: it sometimes doesn't update corectly I think [09:27:21] giving 0 as result [09:27:38] ah didn't get that weird result [09:27:49] so you are doing sum_over_time [5m[ [09:28:08] elukey: I also had a question for you: should we offset that? [09:28:12] correct elukey [09:28:24] what do you mean with offset that? [09:28:52] like, take the sum in between [now-10min, now-5min] [09:29:25] elukey: I wonder about prometheus offset in receiving metrics [09:30:15] like if current minute is always 1 minute late (because currently being ingested), plus some time to actually ingest --> we have 2 minutes delay before correct metric shows up [09:30:38] so maybe we can delay a bit our alerting? [09:30:51] maybe 5 minutes is too long, but 2 or 3 minutes? [09:31:06] and maybe I'm wrong, and promotheus doesn' have delay at all: ) [09:34:16] there is surely ingestion delay (I don't think that prometheus is immune from this problem) but I'd say that we can experiment with this alert and make it publish only to analytics people [09:34:25] then we'll tune it accordingly if necessary [09:35:11] I am wondering though how to read "holes" like https://grafana.wikimedia.org/dashboard/db/prometheus-druid?panelId=41&fullscreen&orgId=1&from=1514929777779&to=1514932307504 [09:35:31] these ones will probably trigger the alarm [09:35:43] are they legitimate or false positives? [09:46:27] elukey: I have actually no idea :( [09:49:08] elukey: maybe summing data for more than 5 minutes is wise? [09:49:52] elukey: while not in FR period, I think the banner activity is a lot smaller, and therefore potentially having zeros [09:49:58] elukey: But I'm not expert [09:53:57] joal: yes I'd be in favor of it, maybe 15/30 mins? [09:54:09] elukey: I agree [09:59:43] elukey: https://gist.github.com/jobar/92f719d6b002d77275f9a48f92297445 [10:03:10] joal: two things! contact_group => 'analytics', dashboard_links => ['https://grafana.wikimedia.org/dashboard/db/prometheus-druid?refresh=1m&panelId=44&fullscreen&orgId=1&from=1514929777779&to=1514932307504'] [10:03:41] indeed elukey !!! [10:13:26] https://blog.wikimedia.org/2018/01/02/wikistats-2/ \o/ [10:13:40] Yessir :) Published yesterday :) [10:15:06] the guy that wrote the article seems a good one [10:15:12] I'd trust him [10:15:13] :D [10:15:37] elukey: Thanks mate :) [10:16:17] elukey: updated the gist: https://gist.github.com/jobar/92f719d6b002d77275f9a48f92297445 [10:17:11] elukey: I have changed the dashboard link to show the processed-evens timelines - I'll probably remove the test-panel once we are done [10:19:29] ack! [10:19:50] I'll add the check after lunch somewhere in puppet and send the code review to you if you want [10:20:02] elukey: awesome [10:20:30] elukey: goig with that one, there is also the cron for restarting yarn job (as a reminder) [10:20:58] yep! [10:41:22] 10Analytics-Kanban, 10Operations, 10ops-eqiad: dbstore1002 possibly MEMORY issues - https://phabricator.wikimedia.org/T183771#3870926 (10Marostegui) I would consider fixing mgmt the first thing address here. If the server breaks, even with OOM, we would need to wait for Chris to reboot it for instance. [10:43:01] 10Analytics-Kanban, 10Operations, 10ops-eqiad: dbstore1002 possibly MEMORY issues - https://phabricator.wikimedia.org/T183771#3870928 (10elukey) >>! In T183771#3870926, @Marostegui wrote: > I would consider fixing mgmt the first thing address here. If the server breaks, even with OOM, we would need to wait f... [10:43:04] wow elukey - Have you seen that vulnerability in intel CPUs? [10:43:23] nope :( [10:43:28] another one? [10:43:46] hm, I think it went public yesteday [10:44:13] elukey: kernel memory vulnerabilit [10:46:05] wow I am reading now [10:46:28] elukey: doesn't look good :( [11:29:08] 10Analytics: Alarm on data quality issues - https://phabricator.wikimedia.org/T159840#3870977 (10mforns) [11:29:10] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Datasets-Webstatscollector, 10Language-Team, and 5 others: Investigate anomalous views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#3870978 (10mforns) [11:29:37] mforns: o/ the alters are running [11:29:58] elukey, awesome! [11:30:17] can you see a speed difference from other hosts? [11:33:30] oh yes :D [11:34:46] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Datasets-Webstatscollector, 10Language-Team, and 5 others: Investigate anomalous views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#3870986 (10mforns) This has become a subtask of T159840. I will change the title from "Inv... [11:36:34] 10Analytics-Data-Quality, 10Analytics-Kanban, 10Datasets-Webstatscollector, 10Language-Team, and 5 others: Add alarms for high volume of views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#3870987 (10mforns) [11:37:54] 10Analytics, 10Analytics-Data-Quality, 10Datasets-Webstatscollector, 10Language-Team: Add alarms for high volume of views to pages with replacement characters - https://phabricator.wikimedia.org/T117945#1787598 (10mforns) [11:41:13] * elukey lunch! [12:39:08] 10Analytics: Enhance mediawiki-history page reconstruction with best historical information possible - https://phabricator.wikimedia.org/T179692#3871108 (10JAllemandou) [12:44:04] 10Analytics: Implement digest-only mediawiki_history_reduced dataset in spark - https://phabricator.wikimedia.org/T181703#3871118 (10JAllemandou) [13:06:14] mforns: about alter speed - a table of 74M records took ~1.5h [13:06:30] elukey, looks good! [13:06:53] the huge tables will take a bit though, it will probably take 2/3 days [13:07:19] IIRC biggest tables were a couple hundreds of millions no? [13:07:26] like 500M? [13:08:15] I think so yes! [13:12:14] so, around 10 hours for those tables I guess... [13:13:30] something like that yes [13:14:11] it is good that db1107 went through a whole run of the cleaner so a lot of tables got reduced [13:14:20] (by DELETE statements) [13:15:32] elukey, makes sense :] [13:16:05] elukey, and even some tables might have been sanitized (updated) because they didn't have non-nullable fields maybe? [13:17:18] yep [13:23:15] * mforns leaves for 90 mins [13:23:27] * elukey feels sad that Marcel leaves [13:24:27] * joal hugs elukey [13:41:01] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx exporter to all the Hadoop daemons - https://phabricator.wikimedia.org/T177458#3871316 (10elukey) New metrics: * HDFS datanode (all hadoop workers) ``` elukey@analytics1049:~$ curl http://analytics1048.eqiad.wmnet:51010/metrics -... [13:41:27] joal: whenever you have time, can you review https://phabricator.wikimedia.org/T177458#3871316 ? [13:41:37] elukey: reading ! [13:42:43] elukey: is there anything in particular I should look for? That's a massive list of metrics :) [13:43:43] joal: nono just if you like naming etc.. this is the list of prometheus metrics :) [13:44:10] I'll ping Filippo to review them but we should be close to start polling them from the prometheus master [13:44:37] after that, jmx trans will probably be discarded [13:44:45] scrape time is now really fast [13:45:09] elukey: \o/ ! [13:46:10] elukey: I wonder why the name in the metric blocks is DataNodeActivity-analytics1029.eqiad.wmnet-50010, and not analytics1029.eqiad.wmnet alone [13:48:23] elukey: same for FSDatasetState-5848a4c5-7a61-4cf0-9577-b88d692c6356 for instance [13:52:33] I think it uses the prometheus agent port [13:52:53] these names are the ones auto-generated by the jmx_agent itself without renaming etc.. [13:53:57] hm, names with GUIDs in them will make it difficult for us to deal with I think :( [13:54:24] I'm ok for verbose DataNodeActivity-analytics1029.eqiad.wmnet-50010, but SDatasetState-5848a4c5-7a61-4cf0-9577-b88d692c6356 is really not self explanatory [13:55:01] well those ones are labels that we probably don't care [13:55:10] Ah, ok |:) [13:55:42] elukey: Except from that, my rapid scan told me there will be plenty stuff we'll be able to do with those new metrics :) [13:55:50] I think so! [13:55:51] :D [14:05:29] 10Analytics, 10Analytics-Wikistats: Questionable metrics from Wikistats 2.0 Alpha - https://phabricator.wikimedia.org/T184011#3869685 (10JAllemandou) Hi! Eyeballing at pageviews for ENWP over the past 2 years (2016 and 2017) in [[ https://stats.wikimedia.org/EN/TablesPageViewsMonthlyCombined.htm | original wik... [14:19:19] HAHA IZER [14:19:30] what an eye elukey ! :p [14:19:51] ottomata: I was like "cooool! Let me steal that stuff!" [14:20:00] :D [14:21:01] hah, you can source my aliases file if you like [14:21:06] it will only have the coolest stuff [14:21:08] aha [14:21:50] elukey: i'm going to merge my patches for cipher suites, etc. [14:22:00] oh after i fix your nit :) [14:25:27] Oh, elukey, did you see this one? [14:25:28] https://gerrit.wikimedia.org/r/#/c/398863/ [14:25:59] that'll let us un-hardcode the kafka brokers for varnishkafka canary [14:26:09] ah yes but forgot to review it, will do it asap [14:30:40] looks good! [14:30:57] let's release it with puppet disabled though, just as precaution [14:32:02] k [14:32:19] i guess on all kafka brokers [14:32:32] hmmm actually this would affect a lot o things [14:32:33] clients too [14:32:49] it mostly won't matter unless somethign is auto subscribed to restart [14:32:55] varnishkafka is i think... [14:32:56] right? [14:32:59] PCC should help? [14:33:11] it definitely helps [14:35:53] ottomata: do you have a min to review https://gerrit.wikimedia.org/r/#/c/401730/3/modules/role/manifests/graphite/alerts.pp ? [14:35:58] I'm around, mforns_away sorry I disappeared yesterday, let's talk wikistats [14:36:02] (whenever you're back) [14:36:13] elukey: with kafka_config.rb change [14:36:13] https://puppet-compiler.wmflabs.org/compiler02/9525/ [14:36:18] no-ops all around [14:36:36] it should be safe, all it does is add entries to the returned $config hash [14:36:42] the existing entries are not modified [14:37:37] you ok If I merge elukey? i dont' rreally want to stop puppet on all hosts that use that function (e.g. all varnishes, etc.) [14:43:11] ottomata: +1 [14:44:52] 10Analytics-Kanban, 10DBA: Purge all old data from EventLogging master - https://phabricator.wikimedia.org/T168414#3871491 (10elukey) Maintenance is ongoing and it will probably last for a couple of days. [14:54:42] joal: still trying to find a good place for the alarm :D [14:58:09] elukey: puppet is disabled on canary, i'm going to manually edit vk and restart there [14:59:30] ack [15:00:05] !log restarting kafka-jumbo brokers to enable tls version and cipher suite restrictions [15:00:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:03:54] elukey: saw your patch - looks good to me :) [15:04:12] the only weird thing is that a prometheus alert is in a graphite role [15:04:31] elukey: I noticed :) [15:04:35] I asked to Filippo what's best for prometheus [15:04:48] elukey: however it fulfill the same global function so .... [15:05:01] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, 10User-Elukey: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3871545 (10Ottomata) > The sigalgs lists being negotiated for mutual certificate-based auth seem to include some weak options Ah I just real... [15:05:17] I'm gonna take a break now - ottomata, elukey: can we move ops-sync an hour later to match standup? [15:05:33] oh [15:05:33] sure [15:05:46] standup on my cal today is still in 1 h though [15:05:50] has that changed? [15:05:54] today? [15:05:58] joal: ^? [15:06:19] ottomata: for me it's in 2 hours [15:07:14] elukey: I wonder about the burrow alarm about EL-mysql being stopped - anyway to prevent it? (I assume no, but prefere to ask :) [15:07:35] oh hm, yeah it is...ok fine w m joal [15:07:36] great [15:07:38] moving it [15:07:46] thanks ottomata :) [15:08:18] joal: we could in theory stop burrow but it will prevent us from seeing other potential problems.. [15:08:32] makes sense elukey :) [15:10:35] ok, later team :) [15:11:11] ottomata: what do you think about rebooting kafka1020 and kafka1022 for kernel + openjdk updates? [15:11:18] (to complete https://phabricator.wikimedia.org/T179943) [15:12:03] sure! [15:12:08] 10Analytics-Kanban, 10Patch-For-Review: Druid Woes - https://phabricator.wikimedia.org/T183273#3871566 (10elukey) [15:12:20] all right [15:12:25] elukey: in the mean time 4.9.65-3+deb9u1~bpo8+1 was uploaded, so let's upgrade the kernel before rebooting [15:12:50] moritzm: sure! let me know when done so I'll proceed [15:13:07] or you wait until next week, then we'll likely have kernels which have the PTI change [15:13:23] actually let's rather do that given how painful the kafka reboots are [15:13:33] ah yes.. another round of reboots /o\ [15:14:58] gotta keep the routine going! [15:15:15] ok so I guess I can close https://phabricator.wikimedia.org/T179943 and then re-open a new one when the new kernel is ready :D [15:15:26] kafka reboots aren't painful, are they? hmmm [15:15:32] elukey: kafka-jumbo has auto rebalance enabled :D [15:15:36] maybe will be less painful... [15:15:57] 10Analytics-Kanban, 10User-Elukey: Restart Analytics JVM daemons for open-jdk security updates - https://phabricator.wikimedia.org/T179943#3871568 (10elukey) We'll have to do another round of reboots probably next week, so the remaining kafka hosts will be done later on. [15:16:02] 10Analytics-Kanban, 10User-Elukey: Restart Analytics JVM daemons for open-jdk security updates - https://phabricator.wikimedia.org/T179943#3871569 (10elukey) [15:17:10] ottomata: yeah but we need to do ALL the analytics hosts again next week :( [15:17:44] http://www.theregister.co.uk/2018/01/02/intel_cpu_design_flaw/ [15:18:46] new year, new kernel to deploy :D [15:20:19] woohoooo [15:23:45] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx exporter to all the Hadoop daemons - https://phabricator.wikimedia.org/T177458#3871588 (10fgiunchedi) >>! In T177458#3871316, @elukey wrote: > New metrics: Looks good to me! [15:25:46] anyone wanna brain-bounce where we should link Wikistats metric definitions to? [15:26:07] because pageviews, uniques, and newly registered have these kinds of links: https://meta.wikimedia.org/wiki/Research:Newly_registered_user [15:26:17] but wikistats metrics are defined in-line for the most part [15:26:56] ooh, I should ask in -research [15:34:04] milimetric, no problem, I hardly worked yesterday after meetings [15:34:16] also, the blog post is already out [15:34:30] oh yeah, but more reason to go fast fast :)_ [15:34:32] but let's select things to do in WS2 [15:34:37] sure [15:34:39] k, cave? [15:34:42] yes [15:40:04] 10Analytics-Kanban, 10Analytics-Wikistats: Replace any debouncing with Vue.nextTick - https://phabricator.wikimedia.org/T180412#3871615 (10Milimetric) a:05Milimetric>03mforns [15:41:46] 10Analytics, 10Analytics-Cluster: Requesting account expiration extension - https://phabricator.wikimedia.org/T183291#3871617 (10Jdcc-berkman) @Nuria OK, I understand those issues. Thank you for taking the time to look through all that. There are a lot of groups in our research community (and the WMF community... [15:44:19] running home, back shortly [15:45:28] 10Analytics-Tech-community-metrics, 10Developer-Relations: Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#3871641 (10Aklapper) [15:46:11] 10Analytics-Tech-community-metrics, 10Developer-Relations: Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#2677473 (10Aklapper) Alright: Not going to remove ** Kill "Changeset Submitters" (`gerrit_top_developers`) list of... [15:49:46] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#3871651 (10Aklapper) [15:52:22] 10Analytics-Tech-community-metrics, 10Developer-Relations (Jan-Mar-2018): Go through default Kibana widgets; decide which ones are not relevant for us and remove them - https://phabricator.wikimedia.org/T147001#3871664 (10Aklapper) 05Open>03declined Changing my mind obviously, as customizing creates mainte... [15:57:35] 10Analytics, 10Analytics-Wikistats: Beta Release: Wikistats: support annotations - https://phabricator.wikimedia.org/T178015#3871688 (10Milimetric) [15:57:37] 10Analytics-Kanban, 10Analytics-Wikistats: Beta Release: Support Annotations on Wikistats 2.0 graphs - https://phabricator.wikimedia.org/T178813#3871690 (10Milimetric) [15:58:07] 10Analytics-Kanban, 10Analytics-Wikistats: Beta Release: Resiliency, Rollback and Deployment of Data - https://phabricator.wikimedia.org/T177965#3871691 (10Milimetric) [15:59:00] 10Analytics, 10Analytics-Wikistats: Beta release: Wikistats: Corners of dashboard miniatures overflow when no data - https://phabricator.wikimedia.org/T178812#3703603 (10Milimetric) fixed, @mforns has priority :) [16:05:45] hellooo [16:06:40] o/ [16:16:44] (03CR) 10Ottomata: [V: 032] Support consumption from multiple topics [analytics/statsv] - 10https://gerrit.wikimedia.org/r/391703 (https://phabricator.wikimedia.org/T179093) (owner: 10Ottomata) [16:24:52] Hea milimetric and mforns :) [16:25:04] y9y9y9 [16:25:14] h3yh3yh3y [16:25:20] h1h1h1 [16:25:25] Have you made decision on docu for WKS2? [16:25:28] no [16:25:38] Wow - milimetric is in numeric mode [16:25:41] 10Analytics, 10Analytics-Cluster: Requesting account expiration extension - https://phabricator.wikimedia.org/T183291#3871793 (10Nuria) @Jdcc-berkman Thanks for taking the time, See for example what a product ionized job looks like here (this is oozie/spark) : https://gerrit.wikimedia.org/r/#/c/383761/ In you... [16:25:43] hey [16:25:46] :d [16:26:33] milimetric: I started a page listing metrics and dimensions for the new metrics this afternoon [16:27:05] milimetric, mforns: It has no point to stay as is, but rather it can be a place where to copy-paste from when we know where we want to document [16:28:02] joal, makes sense [16:28:05] great joal, makes sense [16:28:29] ottomata, you there? :] [16:28:37] (03PS1) 10Ottomata: Have to subscribe if using multiple kafka topics. [analytics/statsv] - 10https://gerrit.wikimedia.org/r/401750 (https://phabricator.wikimedia.org/T179093) [16:29:18] ottomata, do you have 5 mins to talk about eventCapsule in EL :]? [16:29:22] (03CR) 10Ottomata: [V: 032 C: 032] Have to subscribe if using multiple kafka topics. [analytics/statsv] - 10https://gerrit.wikimedia.org/r/401750 (https://phabricator.wikimedia.org/T179093) (owner: 10Ottomata) [16:31:06] mforns: sure! hmm, we are just about to start ops sync [16:31:09] but no one is in bc yet [16:31:11] so you can beat them [16:31:19] no! [16:31:31] mforns: you lose! [16:31:36] xDDD [16:34:30] elukey: saw your new patch for prometheus alerts :) Looks good as well - no longer in graphite :) [16:34:50] joal: so I asked in research, and I'm gonna ping people now because I think they should weigh in [16:35:00] sounds good milimetric [16:35:01] but that's more for the structure [16:35:13] when we decide on that I'll use your content to start [16:35:30] milimetric: so far it's a VERY SMALL content :) [16:35:55] that's fine, it's going to be a lot to write and I'm not looking forward to it [16:36:10] because it's one of those subjects that nobody will agree on the wording [16:36:27] fdans: hola! what is the state of loading the map data in cassandra? [16:36:27] very much agreed [16:37:40] nuria_: holaaaa data is loaded up to December 2017 [16:38:15] fdans: i see, was the loading code merged? [16:38:19] haven't reached out to joal yet about the remaining data, I've been focusing on finishing these couple issues in the vue component [16:38:21] nuria_: yes [16:39:08] fdans, nuria_: data is loaded up to now, and prod job is running [16:41:19] 10Analytics, 10Analytics-Cluster: Requesting account expiration extension - https://phabricator.wikimedia.org/T183291#3871832 (10Jdcc-berkman) @Nuria Sounds good, I'll dig in. I'm familiar with the various RPCA implementations (I wrote the go one), so that part shouldn't be too much trouble. [16:43:46] 10Analytics, 10Analytics-Wikistats, 10Hindi-Sites: Hindi Wikiversity is not showing in Wikimedia Stats - https://phabricator.wikimedia.org/T183682#3860400 (10Milimetric) It looks like there are some reading metrics available, but there seems to be no editing activity whatsoever. That might be wrong, we'll l... [17:00:49] 10Analytics-Kanban, 10Patch-For-Review: Alert on age of backups on analytics1002 - https://phabricator.wikimedia.org/T182327#3871924 (10Nuria) [17:00:53] 10Analytics-Kanban, 10Patch-For-Review: Alert on age of backups on analytics1002 - https://phabricator.wikimedia.org/T182327#3820110 (10Nuria) 05Open>03Resolved [17:01:14] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: wikistats rendering bug - https://phabricator.wikimedia.org/T182817#3871926 (10Nuria) 05Open>03Resolved [17:10:06] 10Analytics-Kanban, 10Analytics-Wikistats: Please add download option 'as csv file' to Wikistats 2 - https://phabricator.wikimedia.org/T183192#3871956 (10Nuria) [17:13:26] 10Analytics-Kanban, 10Patch-For-Review: https://dumps.wikimedia.org/other/pageviews/ needs a README - https://phabricator.wikimedia.org/T167033#3871964 (10Nuria) 05Open>03Resolved [17:30:33] joal: new alarm deployed! [17:30:46] Yay elukey! Thanks a lot :) [17:31:06] elukey: I'll be interested to see if it rings anytime soon [17:31:39] next step https://gerrit.wikimedia.org/r/#/c/395504/ [17:34:55] Gone for diner, back after a-team :) [17:51:01] * elukey off! [18:40:59] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Support multi DC statsv - https://phabricator.wikimedia.org/T179093#3872381 (10Ottomata) Alright! varnishkafka-statsv is producing to main-eqiad Kafka, and hafnium statsv instance is consuming from main-eqiad. You should be able to include `webpe... [18:41:26] 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review: Support multi DC statsv - https://phabricator.wikimedia.org/T179093#3713041 (10Ottomata) [18:51:34] joal: whenever you have a chance, send me those definitions you have, I have to merge them with https://www.mediawiki.org/wiki/Analytics/Metric_definitions [18:52:09] milimetric: I think I'll have something ready for tomorrow [18:52:15] 10Analytics, 10Research: Formal announcement of productized clickstream dataset - https://phabricator.wikimedia.org/T183097#3872416 (10DarTar) [[ https://docs.google.com/document/d/15X-zhhuoixEyryATxnSZC500XWZ5fbl1j04yW8hQzSw/edit?ts=5a4d14c3 | Draft ]] [18:52:31] 10Analytics, 10Analytics-Cluster: Requesting account expiration extension - https://phabricator.wikimedia.org/T183291#3872419 (10Nuria) BTW, superb work on https://dash.harvard.edu/bitstream/handle/1/32741922/Wikipedia_Censorship_final.pdf [19:10:36] joal: fixed the druid prometheus metric, now icinga looks green! [19:10:45] I can confirm that it works fine :) [19:11:57] Heya elukey! I'm sorry I didn't even notice it went red :( [19:12:08] It's great to know it works ! [19:12:21] elukey: I'm gonna push for the other one tomorrow :) [19:12:24] it was unknown, icinga was complaining but not alarming yet :) [19:12:32] yep! [19:12:40] going afk for real now! :) [19:20:58] 10Analytics-Tech-community-metrics, 10Developer-Relations: Investigate listing the "Onboarding New Developers" KPIs on a custom dashboard - https://phabricator.wikimedia.org/T179329#3872611 (10Aklapper) Basic notes for myself how to create a new dashboard: * Go to https://wikimedia.biterg.io/edit/app/kibana#/d... [20:31:12] 10Analytics-Cluster, 10Analytics-Kanban, 10EventBus: Delete stale topics from main Kafka clusters - https://phabricator.wikimedia.org/T149594#3872902 (10Ottomata) [21:26:56] Gone for tonight a-team [21:29:20] laters joal! [21:30:59] 10Analytics-Cluster, 10Analytics-Kanban, 10EventBus: Delete stale topics from main Kafka clusters - https://phabricator.wikimedia.org/T149594#2757362 (10Ottomata) a:03Ottomata Done (mostly!) there are still some weird change-prop retry topics, but for the most part things are much better. We've deleted th... [21:35:15] hmm. I'm blanking. Are we storing (for example, daily) per article pageviews for different Wikipedia languages somewhere on the servers I can easily access? [21:35:34] lzia: yes [21:35:45] lzia: pageview_hourly [21:35:53] lzia: per article per project [21:36:06] lzia: also in pivot [21:36:56] lzia: https://pivot.wikimedia.org/#pageviews-hourly [21:39:17] thanks, nuria_. /facepalm [21:51:55] lzia: vacation rather [21:52:39] as in, the vacation does this to me or I need to go to vacation? cuz for the latter I need to negotiate hard with my manager, given the recent vacation, nuria_. ;) [21:52:40] 10Analytics, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install noteboot100[34] - https://phabricator.wikimedia.org/T183935#3873157 (10Cmjohnson) [22:06:21] lzia: in the same way vacations somehow make you forget your password you know? [22:07:42] 10Analytics-Kanban, 10Patch-For-Review: Remove AppInstallIId from EventLogging purging white-list - https://phabricator.wikimedia.org/T178174#3873179 (10Nuria) I know @APalmer_WMF is busy for I believe there were meetings that already happening, does legal have any guidelines here as to the appinstallId retent... [22:19:19] nuria_: haha. yeah. I was fully dizzy yesterday. jetlag doesn't help either. ;) [22:32:22] (03PS1) 10Nuria: Replacing JSON download with CSV download [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/401814 (https://phabricator.wikimedia.org/T183192) [22:32:50] 10Analytics, 10Analytics-EventLogging, 10Discovery, 10Graphs, and 9 others: RFC: Use YAML instead of JSON for structured on-wiki content - https://phabricator.wikimedia.org/T147158#3873259 (10Krinkle) [22:33:06] (03PS2) 10Nuria: Replacing JSON download with CSV download [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/401814 (https://phabricator.wikimedia.org/T183192) [22:47:48] 10Analytics-Kanban: Remove sensitive fields from whitelist for QuickSurvey schemas (end of Q2) - https://phabricator.wikimedia.org/T174386#3873318 (10Nuria) [22:48:27] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Move away from jmxtrans in favor of prometheus jmx_exporter - https://phabricator.wikimedia.org/T175344#3873320 (10Nuria) [22:51:43] 10Analytics: Investigate oozie suspended workflows - https://phabricator.wikimedia.org/T163933#3215250 (10Nuria) Is this still relevant? [22:52:21] 10Analytics, 10Pageviews-API: Responses on pageview API should be lighter - https://phabricator.wikimedia.org/T145935#2645635 (10Nuria) [22:53:56] 10Analytics-EventLogging, 10Analytics-Kanban: Purge refined JSON data after 90 days - https://phabricator.wikimedia.org/T181064#3873355 (10Nuria) [22:56:02] 10Analytics-Kanban, 10Analytics-Wikistats: Privacy pageview threshold for map report - https://phabricator.wikimedia.org/T181508#3873356 (10Nuria) 05Open>03Resolved [22:56:14] 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: [Wikistats2] Add link path to router-link - https://phabricator.wikimedia.org/T183149#3873358 (10Nuria) 05Open>03Resolved [22:56:43] 10Analytics-Kanban, 10Analytics-Wikistats, 10I18n, 10Patch-For-Review: Move non-SI prefixes to user- or locale-specific interface - https://phabricator.wikimedia.org/T179906#3873361 (10Nuria) 05Open>03Resolved [22:56:54] 10Analytics-Kanban, 10Patch-For-Review: Druid Woes - https://phabricator.wikimedia.org/T183273#3873372 (10Nuria) 05Open>03Resolved [22:57:07] 10Analytics-Cluster, 10Analytics-Kanban, 10Analytics-Wikistats: Add "Pageviews by Country" AQS endpoint - https://phabricator.wikimedia.org/T181520#3873374 (10Nuria) [22:57:09] 10Analytics-Cluster, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Add pageviews by country cassandra oozie job - https://phabricator.wikimedia.org/T181521#3873373 (10Nuria) 05Open>03Resolved [22:57:28] 10Analytics-Kanban, 10User-Elukey: Restart Analytics JVM daemons for open-jdk security updates - https://phabricator.wikimedia.org/T179943#3873390 (10Nuria) 05Open>03Resolved [23:01:06] 10Analytics-Kanban, 10Analytics-Wikistats: Create Daily & Monthly pageview dump with country data and Visualize on UI - https://phabricator.wikimedia.org/T90759#3873409 (10Nuria) [23:02:11] 10Analytics-Kanban, 10Analytics-Wikistats: SEO-friendly HTML titles for Wikistats 2.0 - https://phabricator.wikimedia.org/T182718#3873418 (10Nuria) a:03Nuria [23:21:28] 10Analytics, 10Analytics-Wikistats: Make browser headers information available through Wikistats - https://phabricator.wikimedia.org/T17059#3873498 (10Nuria) 05Open>03Resolved [23:22:02] 10Analytics, 10Analytics-Wikistats: Make browser headers information available through Wikistats - https://phabricator.wikimedia.org/T17059#201059 (10Nuria) This was solved ages ago:https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os [23:23:15] 10Analytics, 10Analytics-Wikistats, 10Design: stats.wikimedia.org is visually unattractive - https://phabricator.wikimedia.org/T28353#3873509 (10Nuria) Solved! http://stats.wikimedia.org/v2 [23:23:19] 10Analytics, 10Analytics-Wikistats, 10Design: stats.wikimedia.org is visually unattractive - https://phabricator.wikimedia.org/T28353#3873511 (10Nuria) 05Open>03Resolved [23:23:42] mforns, fdans , milimetric check this one one out! https://phabricator.wikimedia.org/T28353 [23:24:13] 10Analytics, 10Analytics-Wikistats: Show IE 8 compatibility view separately in SquidReportClients.htm - https://phabricator.wikimedia.org/T26506#3873513 (10Nuria) 05Open>03declined [23:24:45] 10Analytics, 10Analytics-Wikistats: Make non-sensitive browser referrers available through Wikistats - https://phabricator.wikimedia.org/T17060#3873515 (10Nuria) 05Open>03declined [23:25:26] hahaha, nice archive diving nuria_, MZ will like the easter egg :) [23:25:34] 10Analytics, 10Analytics-Wikistats: About the api format type to check the siteinfo - https://phabricator.wikimedia.org/T123702#3873527 (10Nuria) 05Open>03declined [23:26:17] 10Analytics, 10Analytics-Wikistats, 10Browser-Support-Microsoft-Edge: Microsoft Edge user agent is not recognized - https://phabricator.wikimedia.org/T104531#3873531 (10Nuria) Solved: https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-browser [23:26:21] 10Analytics, 10Analytics-Wikistats, 10Browser-Support-Microsoft-Edge: Microsoft Edge user agent is not recognized - https://phabricator.wikimedia.org/T104531#3873532 (10Nuria) 05Open>03Resolved [23:27:33] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#3873533 (10Nuria) I think this ticket can be closed, while redirect might have been valid we do not seem to miss i... [23:27:46] 10Analytics, 10Analytics-Wikistats, 10Operations, 10Regression: [Regression] stats.wikipedia.org redirect no longer works ("Domain not served here") - https://phabricator.wikimedia.org/T126281#3873536 (10Nuria) 05Open>03declined [23:30:12] 10Analytics, 10Analytics-Wikistats: Traffic stats: analyze category 'Linux other' - https://phabricator.wikimedia.org/T48190#3873541 (10Nuria) Closing, stats are available at https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-os but given the shift to our trafic to mobile there is little data... [23:30:35] 10Analytics, 10Analytics-Wikistats: Traffic stats: analyze category 'Linux other' - https://phabricator.wikimedia.org/T48190#3873542 (10Nuria) 05Open>03Resolved [23:31:48] 10Analytics, 10Analytics-Wikistats: Squid log based traffic report SquidReportDevices.htm for mobile devices is broken - https://phabricator.wikimedia.org/T48269#3873547 (10Nuria) Please see: https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-os [23:31:55] 10Analytics, 10Analytics-Wikistats: Squid log based traffic report SquidReportDevices.htm for mobile devices is broken - https://phabricator.wikimedia.org/T48269#3873548 (10Nuria) 05Open>03Resolved [23:32:23] 10Analytics, 10Analytics-Wikistats: iOS stats are wrong in SquidReportOperatingSystems.htm - https://phabricator.wikimedia.org/T65109#3873555 (10Nuria) Please see: https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-os [23:32:31] 10Analytics, 10Analytics-Wikistats: iOS stats are wrong in SquidReportOperatingSystems.htm - https://phabricator.wikimedia.org/T65109#3873556 (10Nuria) 05Open>03Resolved [23:33:02] 10Analytics, 10Analytics-Wikistats: Include version numbers for iPad / iPhone / iPod - https://phabricator.wikimedia.org/T66442#3873562 (10Nuria) https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-os [23:33:07] 10Analytics, 10Analytics-Wikistats: Include version numbers for iPad / iPhone / iPod - https://phabricator.wikimedia.org/T66442#3873563 (10Nuria) 05Open>03Resolved [23:33:31] 10Analytics, 10Discovery, 10EventBus, 10MediaWiki-General-or-Unknown, and 8 others: EventBus MVP - https://phabricator.wikimedia.org/T114443#3873565 (10Krinkle) [23:34:18] 10Analytics, 10Analytics-Wikistats: Modern Fedora distros can not be assessed from user agent string for SquidReportOperatingSystems.htm - https://phabricator.wikimedia.org/T65111#3873577 (10Nuria) Fedora can be seen in our stats but usage given shift to mobile is minimal: https://analytics.wikimedia.org/dashb... [23:34:23] 10Analytics, 10Analytics-Wikistats: Modern Fedora distros can not be assessed from user agent string for SquidReportOperatingSystems.htm - https://phabricator.wikimedia.org/T65111#3873578 (10Nuria) 05Open>03Resolved [23:35:12] 10Analytics, 10Analytics-Wikistats: Traffic stats: start reporting on mobile traffic for non Wikipedia wikis - https://phabricator.wikimedia.org/T48201#3873581 (10Nuria) http://stats.wikimedia.org/v2 has this data [23:35:17] 10Analytics, 10Analytics-Wikistats: Traffic stats: start reporting on mobile traffic for non Wikipedia wikis - https://phabricator.wikimedia.org/T48201#3873582 (10Nuria) 05Open>03Resolved [23:35:49] 10Analytics, 10Analytics-Wikistats: Traffic stats: SquidReportUserAgentsTimed.htm shows wrong report period - https://phabricator.wikimedia.org/T48191#3873585 (10Nuria) New reports: https://analytics.wikimedia.org/dashboards/browsers/#desktop-site-by-os [23:35:57] 10Analytics, 10Analytics-Wikistats: Traffic stats: SquidReportUserAgentsTimed.htm shows wrong report period - https://phabricator.wikimedia.org/T48191#3873586 (10Nuria) 05Open>03declined [23:36:42] 10Analytics, 10Analytics-Wikistats: Different squid log based traffic reports have different %mobile - https://phabricator.wikimedia.org/T48273#3873588 (10Nuria) 05Open>03declined [23:41:22] did EventLogging move from stat1006.eqiad.wmnet to somewhere else ottomata[m] ? [23:41:38] 10Analytics, 10Analytics-Wikistats: Breakdowns should be sticky on url such you can bookmark them - https://phabricator.wikimedia.org/T184136#3873598 (10Nuria) [23:45:43] ottomata[m]: no worries i worked it out :) [23:49:32] 10Analytics, 10Analytics-Wikistats: Wrong y-axis labels on wikistats graph - https://phabricator.wikimedia.org/T184138#3873625 (10Nuria) [23:49:45] 10Analytics, 10Analytics-Wikistats: Wrong y-axis labels on wikistats graph - https://phabricator.wikimedia.org/T184138#3873635 (10Nuria) {F12232779} [23:50:16] 10Analytics, 10Analytics-Wikistats: Wrong y-axis labels on wikistats graph - https://phabricator.wikimedia.org/T184138#3873636 (10Nuria) {F12232781} [23:58:38] 10Analytics: When displaying a graph include metric total not only average - https://phabricator.wikimedia.org/T184139#3873650 (10Nuria) [23:58:49] jdlrobson: ok