[01:43:40] (03PS1) 10Milimetric: Fix copy paste error in sample command [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520351 [01:45:37] (03PS2) 10Milimetric: Fix copy paste error in sample command [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520351 [04:17:33] (03CR) 10Nuria: [C: 03+2] Fix copy paste error in sample command [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520351 (owner: 10Milimetric) [05:57:39] morning! [05:59:00] o/ [06:08:07] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations: Archive zookeeper puppet submodule - https://phabricator.wikimedia.org/T227164 (10elukey) [06:18:08] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations, 10Patch-For-Review: Archive zookeeper puppet submodule - https://phabricator.wikimedia.org/T227164 (10elukey) [06:24:29] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations, 10Patch-For-Review: Archive zookeeper puppet submodule - https://phabricator.wikimedia.org/T227164 (10elukey) There are some pull requests to close in https://github.com/wikimedia/puppet-zookeeper/pulls and also to set the mirror as read only, but... [06:27:16] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations: Archive cdh puppet submodule - https://phabricator.wikimedia.org/T226474 (10elukey) @hashar should I deactivate the repo in diffusion? O do anything else? [07:02:06] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations: Archive cdh puppet submodule - https://phabricator.wikimedia.org/T226474 (10hashar) Yes archive it in Diffusion and we will also just delete the Github mirror. Just a note, it is possible to merge the repository into operations/puppet.git while ke... [07:04:38] 10Analytics, 10Analytics-Kanban, 10Cleanup, 10Operations: Archive cdh puppet submodule - https://phabricator.wikimedia.org/T226474 (10elukey) >>! In T226474#5302653, @hashar wrote: > Yes archive it in Diffusion and we will also just delete the Github mirror. IIUC Timo suggested to leave the github mirror... [07:38:11] ACKNOWLEDGEMENT - Check if the Hadoop HDFS Fuse mountpoint is readable on an-tool1006 is CRITICAL: CRITICAL Elukey Still testing Kerberos configs [08:09:07] 10Analytics, 10Operations, 10Wikimedia-Incident: Move icinga alarm for the EventStreams external endpoint to SRE - https://phabricator.wikimedia.org/T227065 (10MoritzMuehlenhoff) p:05Triage→03Normal [08:17:20] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Enable base::firewall on stat boxes after restricting Spark REPL ports. - https://phabricator.wikimedia.org/T170826 (10elukey) [08:36:50] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Enable base::firewall on stat boxes after restricting Spark REPL ports. - https://phabricator.wikimedia.org/T170826 (10elukey) Found something interesting in https://spark.apache.org/docs/2.3.1/configuration.html#networking ` spark.port.maxRetries 16 Maximum... [08:37:22] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Enable base::firewall on stat boxes after restricting Spark REPL ports. - https://phabricator.wikimedia.org/T170826 (10elukey) p:05Triage→03Normal [08:49:54] 10Analytics, 10User-Elukey: Show IPs matching a list of IP subnets in Webrequest data - https://phabricator.wikimedia.org/T220639 (10elukey) Ack got it thanks! As curiosity I ran two Spark jobs with one hour of webrequest text data and: 1) The Spark python code listed in this task 2) Another similar script... [10:25:33] * elukey errand + early lunch! [11:41:01] 10Analytics, 10Analytics-Kanban: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10JAllemandou) Adding a comment: >>! In T220507#5298048, @Milimetric wrote: > Quick note that we tried to do what we proposed here but it complicated other parts of the data too much. S... [12:55:37] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Modern Event Platform: Stream Configuration Service - https://phabricator.wikimedia.org/T205319 (10Ottomata) @nuria, see comment https://phabricator.wikimedia.org/T205319#5300239. I'm tryin... [13:10:26] (03PS1) 10Fdans: Add interlanguage hadoop queries to queries directory [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/520432 (https://phabricator.wikimedia.org/T222739) [13:15:46] (03PS2) 10Fdans: Limit available granularities to those listed in config [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/518716 (https://phabricator.wikimedia.org/T226397) [13:15:51] (03CR) 10Fdans: "Nuria: yesyes, that's what it's supposed to do, but I think I wrote "yearly" instead of "daily", sorry, just corrected it." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/518716 (https://phabricator.wikimedia.org/T226397) (owner: 10Fdans) [14:16:32] 10Analytics, 10MobileFrontend, 10Readers-Web-Backlog: Having trouble setting up MobileFrontend for development - https://phabricator.wikimedia.org/T226071 (10pmiazga) @Milimetric thanks for the note. It could be something wrong with the env, but we noticed that MediaWiki MobileFrontend/Minerva documentation... [14:33:21] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10elukey) [14:35:26] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10elukey) The /mnt/hdfs mountpoint works with Kerberos, but of course needs the user to be authenticated before reading. There are two use ca... [14:39:01] (03CR) 10Nuria: [C: 03+2] Limit available granularities to those listed in config [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/518716 (https://phabricator.wikimedia.org/T226397) (owner: 10Fdans) [15:08:15] a-team: I'm not feeling well, I'll keep monitoring the job, basically I just have to manually run the checker after the 2019-05 reduced snapshot is done (it's been at 100%map / 100%reduce for a while, but I think disk operations take a long time with the large sizes, should be done sometime tonight). After that we can deploy aqs. [15:10:24] milimetric: take care of yourself :) if you need any help or to tag out I'm here [15:10:40] +1 :) [15:11:07] milimetric: you can share screens over hangouts and I'll look at it while you nap :D [15:12:11] :) no worries, I’m ok enough to log in once in a while. It would be cool if there was like a shared screen so you could pass it off [15:12:50] If anyone wants to run queries against the 2019-06 reduce snapshot, it’d be useful to see that results make sense [15:26:58] AndyRussG: hiya [15:27:03] we'd like to do https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/520019 soon [15:27:07] i guess monday [15:27:15] i don't know how to trigger any of those events tho [15:27:21] i'd like to be able to test that everythin works as expected [15:42:28] 10Analytics, 10EventBus, 10Operations, 10Core Platform Team Backlog (Watching / External), and 2 others: Replace and expand codfw kafka main hosts (kafka200[123]) with kafka-main200[12345] - https://phabricator.wikimedia.org/T225005 (10herron) [15:44:22] 10Analytics, 10Analytics-Cluster, 10User-Elukey: Enable base::firewall on stat boxes after restricting Spark REPL ports. - https://phabricator.wikimedia.org/T170826 (10Ottomata) Also related: T111433 [15:49:02] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10Ottomata) > Also, do we still need to rsync that data? Ya, I believe so: https://dumps.wikimedia.org/other/pageviews/2019/2019-07/ > 2. is... [15:49:23] 10Analytics: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10LGoto) [15:49:50] 10Analytics, 10Reading-Infrastructure-Team-Backlog, 10Epic: Client side error logging production launch - https://phabricator.wikimedia.org/T226986 (10LGoto) [16:00:57] ping fdans [16:24:02] ottomata: interesting, I just created another account and it works [16:24:11] but yours not :D [16:25:06] (03PS1) 10Nuria: Correcting comment [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520488 [16:25:25] fdans: Correcting comment , can you take a look ? https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/520488 [16:26:20] (03CR) 10Fdans: [V: 03+2 C: 03+2] Correcting comment [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520488 (owner: 10Nuria) [16:31:03] same problem with Fran though [16:37:49] ottomata: can you retry now? [16:37:55] (you need to change pw again) [16:38:06] I think it might be a clock skew issue [16:41:35] brb [17:01:49] * elukey off! [17:11:50] elukey: it worked! [17:11:52] clock skew great! :) [17:53:24] (03PS1) 10Nuria: Revert "Correcting comment" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520505 [17:54:59] (03CR) 10Nuria: [C: 03+2] Revert "Correcting comment" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520505 (owner: 10Nuria) [17:56:09] (03CR) 10Nuria: [V: 03+2 C: 03+2] Revert "Correcting comment" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520505 (owner: 10Nuria) [18:00:23] (03CR) 10Nuria: [V: 03+2 C: 03+2] Fix copy paste error in sample command [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520351 (owner: 10Milimetric) [18:07:39] 10Analytics, 10Product-Analytics: Enable widgets on Jupyter Labs on SWAP - https://phabricator.wikimedia.org/T227217 (10nettrom_WMF) [19:10:29] hip: Pchelolo wanna brain bounce some stream config? [19:13:34] 10Analytics, 10OOUI, 10Wikimedia-General-or-Unknown, 10Security: "Sign In" dialog for piwik.wikimedia.org shown when accessing OOUI demos on doc.wikimedia.org - https://phabricator.wikimedia.org/T225882 (10Aklapper) [19:13:51] 10Analytics, 10OOUI, 10Wikimedia-General-or-Unknown: "Sign In" dialog for piwik.wikimedia.org shown when accessing OOUI demos on doc.wikimedia.org - https://phabricator.wikimedia.org/T225882 (10Aklapper) [19:23:55] ottomata: oh sorry man haven't seen your ping [19:24:15] I donno, do you really need me of this thing? it seems much more analytics-specific [19:25:09] hmm, if you can can you help review/brain bounce? this does affect prod stuff e.g. schema/stream mapping configs, it might also have things like topic settings (not sure) [19:25:17] but also ways configurations for stream producers [19:25:22] like remote clients [19:27:49] Pchelolo: ^? [19:28:26] ok, gimme 5 mins towrap up what I'm doing here [19:28:31] k [19:34:38] 10Analytics, 10Analytics-EventLogging, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Modern Event Platform: Stream Configuration Service - https://phabricator.wikimedia.org/T205319 (10Ottomata) [19:35:20] ok ottomata you wanna meet or just chat? [19:35:23] 10Analytics, 10Analytics-Wikistats: X-axis is at odds with stated period in header of trend charts for 'total articles' for a wiki - https://phabricator.wikimedia.org/T180118 (10Blahma) 05Open→03Resolved I suggest marking this a WONTFIX, because it has not even been triaged after almost two years and it ha... [19:35:42] hip: yt still? [19:35:56] ottomata: yup [19:36:08] let's jump in hangout real quick [19:36:12] sounds good [19:36:13] https://meet.google.com/aqg-vebt-mim [19:36:14] ok [19:36:57] 10Analytics, 10Analytics-Wikistats: X-axis is at odds with stated period in header of trend charts for 'total articles' for a wiki - https://phabricator.wikimedia.org/T180118 (10Blahma) 05Resolved→03Open Sorry, I closed this as a mistake. But I am in favor of this being closed as Declined by someone author... [19:38:24] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Cmjohnson) @Ottomata Please decommission the current servers to spare role Please provide the new hostnames... [20:38:58] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10Ottomata) > Please decommission the current servers to spare role Ok will do. I'll downtime the the hostnam... [20:52:16] (03PS1) 10Nuria: 2.6.2 release [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520632 [21:01:03] (03CR) 10Nuria: [C: 03+2] 2.6.2 release [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520632 (owner: 10Nuria) [21:01:19] (03CR) 10Nuria: [C: 03+2] "Self merging per our wikistats deploy protocol" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520632 (owner: 10Nuria) [21:03:24] (03Merged) 10jenkins-bot: 2.6.2 release [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/520632 (owner: 10Nuria) [21:50:34] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats2: Values in map view show unnecessary decimal digits - https://phabricator.wikimedia.org/T200070 (10Nuria) [21:50:45] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Fix status overlay for dates out of bounds - https://phabricator.wikimedia.org/T226402 (10Nuria) [21:51:08] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats UI workarround for time interval bounds - https://phabricator.wikimedia.org/T226421 (10Nuria) [21:54:09] 10Analytics, 10Analytics-Kanban: "All" time range selection should be aware of the metric's available time range - https://phabricator.wikimedia.org/T226486 (10Nuria) [21:57:18] !log deployed wikistats2 https://gerrit.wikimedia.org/r/#/c/analytics/wikistats2/+/520632/ [21:57:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [22:24:02] 10Analytics, 10Analytics-Kanban, 10Operations: Reduce memory allocation for kafkamon instances - https://phabricator.wikimedia.org/T224988 (10Nuria) 05Open→03Resolved [22:24:25] 10Analytics, 10Analytics-Kanban: Refine issues with page links change event - https://phabricator.wikimedia.org/T226268 (10Nuria) 05Open→03Resolved [22:24:38] 10Analytics, 10Analytics-Kanban: Decomission old analytics kafka cluster - https://phabricator.wikimedia.org/T183303 (10Nuria) 05Open→03Resolved [22:47:46] PROBLEM - Check the last execution of reportupdater-interlanguage on stat1007 is CRITICAL: connect to address 10.64.21.118 port 5666: Connection refused [22:48:06] PROBLEM - Check the last execution of reportupdater-browser on stat1007 is CRITICAL: connect to address 10.64.21.118 port 5666: Connection refused [22:49:42] 10Analytics, 10GrowthExperiments, 10Product-Analytics, 10Growth-Team (Current Sprint): Homepage: instrumentation - https://phabricator.wikimedia.org/T216586 (10MMiller_WMF) [22:49:51] 10Analytics, 10Product-Analytics, 10Growth-Team (Current Sprint): Homepage: specify purging strategy - https://phabricator.wikimedia.org/T219252 (10MMiller_WMF) 05Open→03Resolved Thank you, @nettrom_WMF! [22:55:36] PROBLEM - Check the last execution of refinery-import-page-history-dumps on stat1007 is CRITICAL: connect to address 10.64.21.118 port 5666: Connection refused [22:58:14] RECOVERY - Check the last execution of reportupdater-interlanguage on stat1007 is OK: OK: Status of the systemd unit reportupdater-interlanguage [22:58:34] RECOVERY - Check the last execution of reportupdater-browser on stat1007 is OK: OK: Status of the systemd unit reportupdater-browser [23:06:04] RECOVERY - Check the last execution of refinery-import-page-history-dumps on stat1007 is OK: OK: Status of the systemd unit refinery-import-page-history-dumps [23:31:05] hi