[03:08:28] 10Analytics, 10Pageviews-API, 10RESTBase-API, 10Wikifeeds, 10Chinese-Sites: views error in mostread feed - https://phabricator.wikimedia.org/T267624 (10Shizhao) [03:13:59] 10Analytics, 10Pageviews-API, 10RESTBase-API, 10Wikifeeds, 10Chinese-Sites: views error in mostread feed - https://phabricator.wikimedia.org/T267624 (10Shizhao) [07:08:27] good morning [07:25:26] Good morning [07:40:41] !log upgrade hue to hue_4.8.0-2 on an-tool1009 [07:40:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:54:18] joal: still haven't found why pagination doesn't work in hue, really weird [07:54:31] * joal is sadness [07:55:01] elukey: if pagination doesn't work, let's at least remove it, to have access to the needed data :S [07:55:05] not nice either though [07:58:10] nono I'll find the bug, there is also a problem with Hive [07:58:26] :( [08:00:08] but in this case, it is probably because we are running a super old hive [08:00:27] so much time in debugging years old issues [08:01:08] :( [08:01:11] :( [08:01:12] :( [08:01:35] ah no the hive bug is the "usual" one of session expired etc.. [08:01:49] bigtop bigtop bigtop :D [08:09:46] ah joal https://issues.apache.org/jira/browse/BIGTOP-3225 [08:17:20] I don't think I understand --^ correctly elukey - Can you help? [08:18:05] these are the plans for a "cloud native" bigtop in the future, so more kubernetes etc.. friendly [08:18:19] should be good in the future if we decide to use more kubeternetes etc.. [08:18:26] still WIP afaics, but looks promisign [08:18:51] Ack - 'cloud native' words are not really making sense to me plainly :) [08:19:08] joal: see what happens when you use only baremetal for years [08:19:16] :) [08:19:53] * joal is too ol' for those serverless ideas [08:27:09] (03CR) 10Joal: Oozie job for Wikipedia Preview stats (033 comments) [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/635578 (https://phabricator.wikimedia.org/T261953) (owner: 10Sbisson) [08:52:59] btb [08:53:01] *brb [09:29:27] elukey: first successful gobblin job run! Took me a while :) [09:31:52] woooooooooowwww [09:31:56] nice! [09:32:24] elukey: local-mode only, no hourly yet, data not formatted correctly etc - but at least the thing ran :) [09:32:51] joal: yep good first result, it is almost impossible to get everything right at the first attempt [10:21:32] elukey: so the update of 1008 to 5.8 and newe rocm... Should we plan that for Monday? I'd do it on Friday, but... [10:39:03] klausman: hi! I think that even friday is fine [10:41:26] I'm just worried that if things go sideways, everyone will be out :) [10:41:47] On the upside, it being wmf-wide holiday reduces how many people I'll affect [10:43:21] klausman: ah no wait it is wmf holiday, let's do it on monday, sorry I forgot that [10:43:29] my bad [10:43:39] let's not work on holidays :) [10:48:55] Eh, it'd have the mentioned upsides [10:54:24] nono let's plan it for monday [10:55:06] klausman: if you have time next week, I'd ask some help for https://phabricator.wikimedia.org/T254317 [11:00:00] Absolutely [11:05:23] <3 [11:07:06] Will send out the heads up about stat1008 downtime after lunch [11:07:24] (sending the mail is after lunch, the downtime is on Monday :)) [11:14:22] ahhaha yes yes [11:17:37] 10Analytics-Radar, 10Data-release, 10Privacy Engineering, 10Privacy: An expert panel to produce recommendations on open data sharing for public good - https://phabricator.wikimedia.org/T189339 (10Jopparn) >>! In T189339#5606989, @Nuria wrote: > @leila sorry, but we reprioritized this task to be able to wor... [11:20:48] I am bootstrapping an-coord1002 with hive-server2, let's see if it works [11:46:47] seems working so far (tried show databases on an-coord1002's beeline) [11:50:57] lunch! [12:21:59] 10Analytics-Radar, 10Data-release, 10Privacy Engineering, 10Privacy: An expert panel to produce recommendations on open data sharing for public good - https://phabricator.wikimedia.org/T189339 (10JAllemandou) Actually all three have been released, either on dumps and or API or both: #1 Editors by country... [13:23:14] joal: so an-coord1002 is running another hive server 2, with the new analytics-hive.eqiad.wmnet creds [13:23:29] the metastore is still on 1001 [13:24:00] \o/ [13:24:10] elukey: do you wish me to test an oozie job? [13:25:11] joal: yep if you have time it would be great! [13:25:29] elukey: how do you feel about deleting entries on the Maintenance schedule when they are >30 days in the past? [13:25:39] klausman: +1! [13:25:44] Ok, will do. [13:26:09] joal: I'd like to add a high availability metastore setup on the two coords, but probably better to wait for bigtop [13:26:14] to have a more up to date hive [13:26:15] elukey: triple checking - hive-server2 address is jdbc:hive2://an-coord1002.eqiad.wmnet:10000/default [13:26:18] correct? [13:26:45] nope, jdbc:hive2://analytics-hive.eqiad.wmnet:10000/default [13:26:55] elukey: about HA I think it's worth waiting - I have no idea how much o a change it'll be with new hive [13:26:56] otherwise we cannot failover transparently [13:27:02] MEH [13:27:07] * joal is dumb [13:27:18] joal: my fear is that the current version doesn't support it correctly :( [13:27:27] very possible elukey [13:27:44] elukey: devising a small test with new metastore [13:27:50] elukey: first, manual test [13:29:51] then we can change one production job, and kick it off with the new settings [13:30:54] !log add hive-server2 to an-coord1002 [13:30:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:33:56] elukey: fisrt try using beeline - ok! [13:34:51] joal: one thing - the hive-site.xml config is not deployed everywhere, only on an-coord1002 [13:35:00] hm [13:35:03] so in other places you'll pick up the an-coord1001 config [13:35:13] I cannot change it everywhere yet [13:35:27] ok course - I don't think it matters [13:38:01] 10Analytics-Clusters, 10Patch-For-Review: Review an-coord1001's usage and failover plans - https://phabricator.wikimedia.org/T257412 (10elukey) I added another hive-server2 to an-coord1002, for the high availability metastore I'd wait until after bigtop to have a more up to date version of hive to work on. Me... [13:40:39] 10Analytics, 10Analytics-Wikistats, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10Isaac) > I'll be looping in a privacy engineer very soon, and I'm interested in hearing their opinion regarding the bread... [14:05:15] 10Analytics, 10Analytics-Kanban: Set up automatic deletion/snitization for netflow data set in Hive - https://phabricator.wikimedia.org/T231339 (10mforns) This was super helpful, @ayounsi, thanks! [14:16:58] going to take a quick walk outside until there is some light, be back in ~30 mins [14:48:06] !log restarted data quality stats daily bundle with new metric [14:48:07] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:53:12] back! [15:37:07] 10Analytics, 10Better Use Of Data, 10Event-Platform: Adopt conventions for server receive and client/event timestamps in non analytics event schemas - https://phabricator.wikimedia.org/T267648 (10Ottomata) [15:37:25] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10MW-1.35-notes (1.35.0-wmf.37; 2020-06-16), 10Patch-For-Review: Clients need to generate an ISO 8601 formatted timestamp - https://phabricator.wikimedia.org/T240460 (10Ottomata) [15:39:01] 10Analytics, 10Better Use Of Data, 10Event-Platform: Adopt conventions for server receive and client/event timestamps in non analytics event schemas - https://phabricator.wikimedia.org/T267648 (10Ottomata) [15:59:40] joal: I'm struggling with some spark, if you're around I could use a nudge [16:06:31] anyone else? :) [16:06:46] milimetric: can I help? [16:06:48] milimetric: ? [16:06:49] depends on what you are doing! [16:06:58] ah [16:07:50] ottomata: sure, that'd be great, omw cave [16:08:14] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10MW-1.35-notes (1.35.0-wmf.37; 2020-06-16), 10Patch-For-Review: Clients need to generate an ISO 8601 formatted timestamp - https://phabricator.wikimedia.org/T240460 (10jlinehan) >>! In T240460#6616381, @Ottomata wrote: > @jlinehan, I'll let you or @Mh... [16:16:01] 10Analytics, 10Operations: Augment NEL reports with a computed timestamp-of-generation - https://phabricator.wikimedia.org/T266886 (10jijiki) p:05Triage→03Medium [16:31:13] (03PS1) 10Fdans: Add Japanese, Italian and Portuguese (Portugal) i18n to Wikistats [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/640466 [16:32:15] (03CR) 10Fdans: [V: 03+2 C: 03+2] "self-merging for deployment" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/640466 (owner: 10Fdans) [16:33:20] joal: just added wikistats deployment to the train etherpad, with 2 checks that the deployment has been successful, let me know if you have any questions :) [16:50:55] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to production shell groups for JAnstee - https://phabricator.wikimedia.org/T266249 (10jijiki) 05Open→03Stalled [16:56:09] 10Analytics, 10Operations, 10SRE-Access-Requests: Requesting access to production shell groups for JAnstee - https://phabricator.wikimedia.org/T266249 (10jijiki) 05Stalled→03Open [16:57:06] Heya team [17:01:41] yoohoo fdans :) [17:01:43] mforns: [17:01:49] uop! [17:16:11] 10Analytics-Radar, 10Operations, 10ops-eqiad: analytics1046/analytics1057 stuck in booting - https://phabricator.wikimedia.org/T267392 (10Cmjohnson) Both servers are stuck at the same spot during post. I tried rebooting an-1046 but it still sticks, One of the power supplies is bad and I replaced it with one... [17:22:49] 10Analytics-Radar, 10Operations, 10ops-eqiad: analytics1046/analytics1057 stuck in booting - https://phabricator.wikimedia.org/T267392 (10Cmjohnson) @elukey @razzi @wiki_willy The servers are stuck and I cannot update bios or firmware. Please decommission. [17:24:51] 10Analytics-Radar, 10Operations, 10ops-eqiad: analytics1046/analytics1057 stuck in booting - https://phabricator.wikimedia.org/T267392 (10elukey) 05Open→03Resolved Thanks for checking @Cmjohnson, will do :) [17:47:12] 10Analytics: Check data currently stored on thorium and drop what it is not needed anymore - https://phabricator.wikimedia.org/T265971 (10elukey) I started a copy of /srv/backup to stat1004 via transfer.py, so we'll be able to push the backup to hdfs. [17:57:23] (03CR) 10Joal: [C: 03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638021 (https://phabricator.wikimedia.org/T267009) (owner: 10Joal) [18:03:17] (03Merged) 10jenkins-bot: Add caching to maxmind readers in core package [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/638021 (https://phabricator.wikimedia.org/T267009) (owner: 10Joal) [18:11:17] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10Cmjohnson) @elukey Let's schedule this for next Tuesday please 1500UTC (10EST) [18:12:00] (03PS1) 10Joal: Update changelog.md for version v0.0.139 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/640478 [18:12:21] anyone for a quick check please ? --^ [18:14:46] ok, merging myself :) [18:15:12] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for dpeloy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/640478 (owner: 10Joal) [18:16:56] !log Releasing refinery-source v0,0,139 to archiva [18:16:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:18:15] ottomata: for eventstreams ui, is browsing streams a use case, or would the user always know which stream to choose? Also, would the user want to know about the stream config? [18:18:40] fdans: Heya quick question on deploying wikistats [18:19:13] fdans: should I bump a minor version or a patch one? [18:19:24] possibly milimetric knows as well --^ [18:19:35] joal patch :) [18:19:41] thanks mate :) [18:24:36] fdans: I get an error at npm install :( [18:24:45] fdans: I think it's becasue of my node versions [18:27:25] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: 2020-09-15) upgrade/replace memory in stat100[58] - https://phabricator.wikimedia.org/T260448 (10elukey) >>! In T260448#6617045, @Cmjohnson wrote: > @elukey Let's schedule this for next Tuesday please 1500UTC (10EST) Looks good for me,... [18:27:50] fdans: problem solved - my npm was outdated - sorry for the noise [18:32:15] hm - now I'm facing something else in weirdness - fdans would you be here? I'm asuming you're preping food maybe? [18:33:29] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10elukey) Sadly 1046 and 1057 need to be decommissioned. At this point, with 14 "old" workers remaining (not sufficient for our purposes), I think it is better to just decom all of th... [18:34:24] joal: maybe I can help [18:34:31] Omw cave [18:34:36] thanks milimetric - omw [18:43:30] 10Analytics-Clusters, 10Patch-For-Review: Create a temporary hadoop backup cluster - https://phabricator.wikimedia.org/T260411 (10Ottomata) +1 [18:55:09] mforns: browsing is a use case [18:55:15] stream config, i think not. [18:55:21] a stream config gui would be cool [18:55:21] but [18:55:28] i don't think eventstreams is the place for it [18:55:31] mayyyybe [18:55:32] maybe [18:55:37] but that's not what we are going for atm [18:55:37] anywa [18:55:38] y [18:56:05] ottomata: ok, are streams going to be categorized, or is it just a flat list? [18:56:34] flat list [18:56:45] multiple can be selected [18:56:50] oh! [18:56:51] although i don't expect people to do that much [18:57:14] mforns: in case you don't have the link [18:57:19] some hacky example code is here [18:57:19] https://codepen.io/ottomata/pen/LYpPpxj?editors=1010 [18:59:02] is streaming more than one strem on the same tab useful? [19:02:39] ottomata: ^ [19:02:54] mforns: i think so [19:03:02] maybe you want to see both page creates and deletes [19:03:13] aha, makes sense! [19:03:15] or maybe you want to see both kaios client errors and mediawiki client errors [19:03:24] understand [19:04:32] ottomata: in case of a replay, what do you think is the best way to limit the events: i.e. by start dt and #max evts.? [19:04:41] or by start dt and end dt? [19:05:48] hm, not sure I understand mforns [19:06:23] if you wanted to replay events from say last night, what would you like to do: give the start dt and the number of events you want to replay? [19:06:35] or give the start dt and the end dt? [19:06:39] hm [19:06:57] the only use case i had in mind was viewing the stream [19:07:02] or relative time periods? [19:07:03] giving a since param would be a timestamp [19:07:28] that would start consuming the stream from that timestamp onward [19:07:34] yea [19:07:41] perhaps pausing the stream is useful [19:07:45] aha [19:08:09] * elukey ofk! byeee [19:08:15] byeeeee [19:32:02] (03PS1) 10Joal: Release 2.8.2 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/640507 [19:32:16] !log Deploy wikistats2 v2.8.2 [19:32:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:32:58] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for dpeloy" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/640507 (owner: 10Joal) [19:34:17] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.0.139 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/640508 [19:36:59] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for new jars" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/640508 (owner: 10Maven-release-user) [19:37:25] mforns: refinery v0,0,139 deployed :) [19:37:35] joal: :D thanks! [19:45:30] fdans: wikistats2 dpeloyed, changes confirmed :) [19:50:31] And docs updated: https://wikitech.wikimedia.org/w/index.php?title=Analytics%2FSystems%2FWikistats_2&type=revision&diff=1887778&oldid=1874914 [19:50:46] Ok done for tonight [19:50:53] Gone for dienr [19:51:02] Enjoy your time off team :) [19:59:30] mforns: i might have a way for you to get the stream names directly from the spec [19:59:41] ottomata: aha! [19:59:42] rather than having to parse it out of the paths like /v2/stream/revision-create [19:59:58] i'm doing some fancy stuff with stream config to dynamically generate the spec better [20:00:04] ok [20:00:11] the declared route /v2/stream/{stream} [20:00:38] will have the stream parameter documented and schemaed with an openapi schema [20:00:50] it'll have an enum with the possible balues [20:00:51] values [20:00:59] so you could parse it directly from the spec json [20:01:07] ok [20:01:10] or who knows, maybe a swagger/openapi library would give you some nice API to it [20:01:23] parsing is fine [20:01:39] patch soon, just gotta make tests run.... [20:01:50] :] [20:16:24] mforns_brb: when you back want brain bounce. :) [20:16:29] i want* [20:19:26] joal: sorry, was indeed making lunch and then plunged into a one on one. Thank you so much Joseph <3 [20:25:37] ottomata: back! [20:26:45] mforns: i got my brain bounce from Pchelolo in -services :p [20:26:57] ok [22:42:08] byeeeeeee!