[05:13:28] 10Analytics, 10Operations, 10ops-codfw: furud mgmt interface is down - https://phabricator.wikimedia.org/T252616 (10Marostegui) [05:13:45] 10Analytics, 10Operations, 10ops-codfw: furud mgmt interface is down - https://phabricator.wikimedia.org/T252616 (10Marostegui) p:05Triage→03Medium [06:36:48] 10Analytics: Use types in Analytics Puppet classes/profiles/etc.. - https://phabricator.wikimedia.org/T252617 (10elukey) [06:37:37] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Unify stat1007 puppet role with the rest of the stats cluster - https://phabricator.wikimedia.org/T249754 (10elukey) This is done now, further refinements might be needed but I'd say that this task can be closed! [06:37:44] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Unify stat1007 puppet role with the rest of the stats cluster - https://phabricator.wikimedia.org/T249754 (10elukey) [06:40:11] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10elukey) @Nuria I am wondering what is best for this task. The ticket last 48h now, but we haven't thought about any hard ru... [06:40:57] (03PS37) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [06:41:03] 10Analytics, 10Analytics-Kanban: Request a Kerberos identity for jrobson - https://phabricator.wikimedia.org/T252222 (10elukey) [06:41:12] 10Analytics, 10Analytics-Kanban: Request a Kerberos identity for jrobson - https://phabricator.wikimedia.org/T252222 (10elukey) 05Open→03Resolved p:05Triage→03Medium [06:41:46] 10Analytics, 10Analytics-Kanban: Kerberos-run-command doesn't work with spark-submit [workaround] - https://phabricator.wikimedia.org/T250161 (10elukey) [06:47:33] !log upgrade spark2 on stat1004 - canary host - T250161 [06:47:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:47:36] T250161: Kerberos-run-command doesn't work with spark-submit [workaround] - https://phabricator.wikimedia.org/T250161 [06:47:41] fdans: --^ [06:48:40] elukey: hell yea, nice :D [06:48:45] just tried sudo -u analytics-privatedata kerberos-run-command analytics-privatedata spark2-submit and the exec format error is gone :) [06:49:17] great, that will make it way nicer to ingest EL data into druid for sure [06:49:22] thank you Luca :) [06:49:23] 10Analytics, 10Analytics-Kanban: Kerberos-run-command doesn't work with spark-submit [workaround] - https://phabricator.wikimedia.org/T250161 (10elukey) Just tried `sudo -u analytics-privatedata kerberos-run-command analytics-privatedata spark2-submit` on stat1004 and the exec format error is gone. Let's test... [06:52:00] 10Analytics: Address refinery security vulnerabilities with jackson and netty - https://phabricator.wikimedia.org/T237774 (10elukey) @Nuria can this be moved to ops week so we can fix the deps? Seems to be something not super pressing but I'd prefer to remove those warnings just to be sure :) [06:59:55] (03PS38) 10Fdans: Add pageview daily dump oozie job to replace Pagecounts-EZ [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) [07:00:16] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) Users still having notebooks on 1003: ` addshore andrew andyrussg awight bearloga conniecc1 dcausse dr0ptp4kt dsaez ebernhardson eyener fdans fsalutari gilles halfak iflorez isaacj jdl jiawang jkumalah joal kartik lad... [07:02:52] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) [07:03:32] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) [07:07:16] remember kids, oozie subworkflows have different workflow ids to their parent workflows [07:07:23] 🤦🏻‍♂️ [07:12:34] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) [07:12:51] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) Pinging people explicitly to set a reminder (15 days before the deprecation): @Addshore @AndyRussG @awight @mpopov @cchen @dcausse @dr0ptp4kt @diego @EBernhardson @eyener @fdans @Fsalutari @Gilles @Halfak @Iflorez @Is... [07:45:36] (03PS1) 10Awight: Count displayed rows [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) [08:00:16] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10KartikMistry) @elukey I'm not using notebook100* Feel free to delete it. [08:09:19] joal: The missing feature to make Vegas safe for public notebooks: https://github.com/vegas-viz/Vegas/issues/129 [08:26:04] Vega has clearly won the dataviz game :) [08:35:25] (03CR) 10Thiemo Kreuz (WMDE): "I never worked with .scala files before, so what I have are really only curious questions, not meant to judge the code or anything." (033 comments) [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [08:41:51] milimetric: early start of the day? :D [08:53:51] (03CR) 10Awight: ">" (033 comments) [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [08:57:18] 10Analytics, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10elukey) Before starting, there are some notes to keep in mind: - we have currently 6 nodes running cassandra 2.2 - 3 of them are due to be refreshed due to hw warranty expiration - we h... [08:58:34] dear cassandra, here we are again [09:21:45] (03CR) 10Thiemo Kreuz (WMDE): Count displayed rows (032 comments) [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [09:28:42] joal: when you have a moment - let's discuss how many nodes we'll need for cassandra [09:39:48] (03CR) 10Fdans: "Applied approach discussed with @Joal, tested correctly, ready for review now for realskies." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) (owner: 10Fdans) [10:04:44] 10Analytics, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10elukey) IIUC, due to hw budget changes, we have to replace all the 6 nodes sometimes in 2021, so the in place upgrade option seems not worth it (will need to confirm this, just sent an e... [10:10:09] 10Analytics, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10elukey) As far as timing, I see from https://cassandra.apache.org/download/ that Cassandra 2.2 will be EOLed when 4.0 will be out (no release date yet). To avoid rushing, I'd start the w... [10:28:44] * elukey lunch + errand! [10:32:15] joal: yesssss [10:32:18] https://www.irccloud.com/pastebin/72Im6osT/ [10:53:15] (03CR) 10Awight: Count displayed rows (032 comments) [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [11:13:50] fdans: cool! :D, what is the ID? [11:14:18] mforns: the id? [11:14:30] the A3B9C2D6E3F5G5H4J4K47L18N1O4P3Q6R3S3T3U1V4W1X1 [11:14:45] mforns: oh that's the hourly data [11:14:57] the letters correspond to the hours of the day [11:15:11] the numbers following them are the hourly views [11:15:16] oooohh [11:15:36] it's pagecounts-ez's hourly format invented by ezachte [11:15:41] ok ok [11:15:46] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10Ladsgroup) >>! In T249752#6132127, @KartikMistry wrote: > @elukey I'm not using notebook100* Feel free to delete it. Same here, thanks! [11:16:04] cool UDTF! [11:16:05] mforns: this UDTF explodes it into hourly rows, adjusting the skewed values [11:17:16] I understand now, great! [11:40:18] \o/ fdans :) [11:40:28] looks great! [11:40:49] elukey: Here I am, please ping when you want about cassandra [11:41:35] (03PS1) 10Awight: Initial vega graphs for row count [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 [11:45:51] (03CR) 10Awight: "Oof, sorry about the near-binary file, I don't think it's possible to omit the source rows for these visualizations :-/" [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (owner: 10Awight) [11:54:41] (03PS2) 10Awight: Count displayed rows [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) [11:54:44] (03PS2) 10Awight: Initial vega graphs for row count [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507) [11:54:47] (03PS1) 10Awight: Move sample window to April [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596192 (https://phabricator.wikimedia.org/T252507) [12:01:19] (03CR) 10Joal: [C: 04-1] "Minor comments on naming and one error in data dependency" (0310 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/595152 (https://phabricator.wikimedia.org/T251777) (owner: 10Fdans) [12:06:31] (03CR) 10Awight: "We'll have to find a better workflow, but to preview the notebook you can either install nbviewer, jupyter, or visit this link: https://nb" [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [12:06:47] thank you for the review joal [12:07:03] np fdans :) I'm trying not to be too late :) [12:07:12] mforns: I'm currently reading yours :) [12:17:33] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10diego) Please feel free to delete it. [12:20:25] (03CR) 10Joal: "Some comments, nothing major" (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/595189 (https://phabricator.wikimedia.org/T251542) (owner: 10Mforns) [12:26:29] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10awight) I've copied and deleted my files. Thanks! [12:26:52] 10Analytics, 10Operations, 10Traffic, 10User-jbond: Fix geoip updaters for new MaxMind hashed keys by 2019-08-15 - https://phabricator.wikimedia.org/T228533 (10Marostegui) It's been around 10 months since the last update, anything pending here? [12:48:05] joal: o/ [12:48:27] Hi elukey [12:48:34] just came back [12:49:12] :) [12:49:31] do you have 5 mins now? [12:49:36] I do :) [12:50:09] bc? [12:50:19] yup [13:16:02] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10JAllemandou) I'd rather hard-code dataset folder to eqiad and read from both. Job will fail with SLAs when DC-switch happen, and we'll hav... [13:16:46] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10Ottomata) In newpyter we will likely try some idle notebook culler, so if that is what you are thinking of, we can do it as... [13:32:21] elukey: question ab [13:32:27] about notebook migration sorry [13:32:43] which host should I target? is stat1004 fine? or should I go to stat1008? [13:33:22] wasn't stat1004 orignially created so researchers would have a place that was not busy stat1007? [13:33:30] joal: i'd go for 1007 or 1008 or even 1005 :) [13:34:38] ottomata: stat1004 is less powerful than others - it was a pure hadoop-edge IIRC [13:35:44] (03PS3) 10Awight: Count displayed rows [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) [13:36:44] (03PS3) 10Awight: Initial vega graphs for row count [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507) [13:43:19] (03CR) 10Thiemo Kreuz (WMDE): "I would give this a blind +2 without really knowing what it does. But unfortunately the commit message is very short and does not really e" [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [13:44:32] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Move sample window to April [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596192 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [13:46:16] joal: thanks for the review [13:46:45] !log upgrade spark2 on all stat100x hosts - T250161 [13:46:48] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:46:48] T250161: Kerberos-run-command doesn't work with spark-submit [workaround] - https://phabricator.wikimedia.org/T250161 [13:50:19] 10Analytics: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster - https://phabricator.wikimedia.org/T252675 (10Ottomata) [13:51:56] 10Analytics, 10Operations, 10ops-codfw: furud mgmt interface is down - https://phabricator.wikimedia.org/T252616 (10Papaul) a:03Papaul [13:55:18] 10Analytics: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster - https://phabricator.wikimedia.org/T252675 (10elukey) One thing - 1007-9 have buster, so we'll need to adjust puppet to deploy openjdk-8 instead of 11. After this I think it should be a simple apply 1. and 2. :) Second thin... [13:56:06] 10Analytics: Analytics Hardware for Fiscal Year 2019/2020 - https://phabricator.wikimedia.org/T244211 (10elukey) [13:59:53] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10EYener) I've deleted my notebooks as well. Thank you! [14:09:59] (03PS1) 10Awight: Remove incomplete queries [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596224 [14:10:01] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Kerberos-run-command doesn't work with spark-submit [workaround] - https://phabricator.wikimedia.org/T250161 (10elukey) [14:10:04] (03PS1) 10Awight: [WIP] Clean up table persistence [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 [14:15:44] (03PS4) 10Awight: Analyze row count [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507) [14:17:20] (03CR) 10Awight: "I can get into more detail here, but it might make more sense to discuss details of the investigation in Phabricator." [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596188 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [14:17:23] 10Analytics: SQL query failed on superset SQL lab - https://phabricator.wikimedia.org/T252225 (10Nuria) I tried this query in Presto and (with modifications) works but for any presto standard is quite slow, due to the factors that @JAllemandou has mentioned so while it finally works it takes about 1-2 minutes a... [14:17:47] 10Analytics: Address refinery security vulnerabilities with jackson and netty - https://phabricator.wikimedia.org/T237774 (10Nuria) Agreed, moved to ops week [14:22:26] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10Nuria) I think is fine to close knowing that tickets expire at 48h . This in fact means that the notebook 'dies' after 48 h... [14:27:43] 10Analytics: Refine event pipeline at this time refines data in hourly partitions without knowing if the partition is complete - https://phabricator.wikimedia.org/T252585 (10Nuria) >I'd rather hard-code dataset folder to eqiad and read from both. Job will fail with SLAs when DC-switch happen, and we'll have to... [14:33:37] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) We talked about grouping by project rather than country to be able to count bots that are doing edits located in labs. We also... [14:35:38] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) Some work here: https://docs.google.com/spreadsheets/d/1GzyDzCuOAjEU6sF3Gs0fiPhGXvZxGYJ4_VVWt07hMpY/edit#gid=731709624 [14:52:16] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10mpopov) emptied out /home/bearloga and shut my jupyter process down. thanks! [14:56:51] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10nshahquinn-wmf) >>! In T246132#6133625, @Nuria wrote: > I think is fine to close knowing that tickets expire at 48h . This... [15:01:56] (03PS2) 10Milimetric: [WIP] Use new page move incremental updates [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) [15:03:43] (03PS2) 10Awight: [WIP] Clean up table persistence [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 [15:04:45] (03CR) 10Awight: Count displayed rows (031 comment) [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596145 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [15:08:34] joal: I missed your question about notebooks, https://wikitech.wikimedia.org/wiki/Analytics/Systems/Clients has a list of stats with their cps/memory/etc.. [15:08:47] no preference, 1008 and 1005 have buster (so more up to date stuff) [15:08:50] thanks elukey :) [15:09:55] ah and they have a gpu of course [15:27:54] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+2] Remove incomplete queries [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596224 (owner: 10Awight) [16:00:00] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10Nuria) For everyone on this list (myself included) if you do not use the notebooks please be so kind to empty your home dir in both machines, thank you. [16:03:02] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10elukey) It is fine also to leave files in there, just be mindful that at some point I'll decom the host and they'll be not accessible anymore :) [16:24:42] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10Halfak) No concerns from me. Feel free to delete. [16:25:52] 10Analytics, 10Cassandra, 10User-Elukey: Cassandra3 migration plan proposal - https://phabricator.wikimedia.org/T249756 (10Eevans) >>! In T249756#6132278, @elukey wrote: > Before starting, there are some notes to keep in mind: > > - we have currently 6 nodes running cassandra 2.2 > - 3 of them are due to be... [16:27:27] * elukey off! [16:27:28] o/ [16:29:18] (03CR) 10Awight: [V: 03+2] Move sample window to April [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596192 (https://phabricator.wikimedia.org/T252507) (owner: 10Awight) [16:29:25] (03CR) 10Awight: [V: 03+2] Remove incomplete queries [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596224 (owner: 10Awight) [16:34:45] (03CR) 10Awight: [C: 04-1] "Grr, a million kwH later the lazy-loading isn't working..." [analytics/wmde/TW/edit-conflicts] - 10https://gerrit.wikimedia.org/r/596225 (owner: 10Awight) [16:47:29] 10Analytics, 10Patch-For-Review: Add new kafka brokers kafka-jumbo100[789] to the jumbo-eqiad Kafka cluster - https://phabricator.wikimedia.org/T252675 (10Ottomata) Before doing any migration, I'm going to some delete old/temp/unused/empty topics: ` -l USER_REVISION_CREATES_PER_DOMAIN_PER_SESSION_OTTO1 USER_R... [16:50:58] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10Fsalutari) I have deleted all my files, thank you! [17:04:39] (03PS3) 10Milimetric: Use new page move incremental updates [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) [17:04:48] joal: if you have a sec, this is with hardcoded eqiad ^ [17:05:01] ack milimetric - reading [17:19:57] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) Some thoughts/hyphotesis: the ratio of bots/(active editors) will be high in emerging communities but low on stablished communi... [17:29:50] 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Epic: Spark sessions can provision kerberos tickets in a more predictable manner - https://phabricator.wikimedia.org/T246132 (10Nuria) [17:29:54] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10Nuria) [17:55:17] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10Isaac) All files cleared -- thanks! [17:57:03] 10Analytics, 10Operations, 10decommission, 10ops-eqiad: Decommission analytics100[1,2] - https://phabricator.wikimedia.org/T205507 (10Cmjohnson) 05Open→03Resolved removed from rack, verified DNS, and switch ports were removed. [17:57:07] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Replace the Analytics HDFS/Yarn masters (hardware refresh) - https://phabricator.wikimedia.org/T203635 (10Cmjohnson) [17:57:52] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10SNowick_WMF) Hi- I'm moving my published reports to stat1007 and don't have write permission to the /srv/published directory, can you check that we have that, and for the other stat servers as well? Is that a special permissio... [18:04:01] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10S.piccardi) Il 13/05/20 18:03, elukey ha scritto: > View Task > elukey added a comment. > > > It is fine also to leave files in there, just be mindful that at some > point I'll dec... [18:05:15] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10RhinosF1) Per above, removed Simon from subscribers to this task. [18:08:37] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue) - https://phabricator.wikimedia.org/T252703 (10soworu) [18:37:53] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue) - https://phabricator.wikimedia.org/T252703 (10Nuria) @soworu please have your manager approve this request [18:41:43] 10Analytics, 10Analytics-Kanban, 10LDAP-Access-Requests, 10Operations: LDAP access to the wmf group for Segun Oworu (superset, turnilo, hue) - https://phabricator.wikimedia.org/T252703 (10ahemmer) Hi @Nuria Approved for @soworu. Thank you! [18:47:51] Woopsy we have forgotten to move AQS druid snapshot :S Just posted a CR [19:16:18] (03CR) 10Joal: "Minor comment on naming, plus one on dependency period" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/594719 (https://phabricator.wikimedia.org/T249773) (owner: 10Milimetric) [19:39:54] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10TJones) All cleared out. [20:00:34] 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10dr0ptp4kt) Deleted things, stopped notebooks and terminals, stopped my Jupyter server. So long notebook1003, it was nice knowing you. [20:30:20] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10Ottomata) In a meeting with @SNowick_WMF @nshahquinn-wmf and @nettrom_WMF today, Neil pointed out a potential problem with the HDFS contents manager approach for notebook files.... [20:43:28] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10diego) @Ottomata, what about the data living in HDFS and the code living in a shared folder mounted through sshfs (or similar). I understand that might be some security issues w... [20:47:39] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10Ottomata) Hm, @diego I think the problem is more complex than that. The intention is to run notebooks in Yarn, which means that your Notebook Server will run on any Hadoop work... [21:03:01] PROBLEM - Hadoop DataNode on analytics1055 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.64.5.18: Connection reset by peer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:03:01] PROBLEM - Disk space on Hadoop worker on analytics1055 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.64.5.18: Connection reset by peer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:03:11] PROBLEM - Hadoop NodeManager on analytics1055 is CRITICAL: CHECK_NRPE: Error - Could not connect to 10.64.5.18: Connection reset by peer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:31:00] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10diego) @Ottomata, the sshfs will go to a centralized (NAS-like) ~/ folder, with small quota just to store the code (very similar with what you would store on a git repository).... [21:36:22] !log powercycle analytics1055 [21:36:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:38:16] RECOVERY - Disk space on Hadoop worker on analytics1055 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:38:16] RECOVERY - Hadoop DataNode on analytics1055 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:38:29] \o/ [21:38:32] RECOVERY - Hadoop NodeManager on analytics1055 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Administration [21:44:16] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Newpyter - SWAP Juypter Rewrite - https://phabricator.wikimedia.org/T224658 (10Ottomata) I see ya, so just using sshfs instead of NFS. Aye either would be functionally equivalent, although I'd expect implementing that securely with sshfs in our stack woul... [21:55:19] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Evaluate possible replacements for Camus: Gobblin, Marmaray, etc. - https://phabricator.wikimedia.org/T238400 (10Ottomata) a:03Ottomata [21:56:17] 10Analytics, 10Analytics-Kanban, 10Event-Platform: Evaluate possible replacements for Camus: Gobblin, Marmaray, etc. - https://phabricator.wikimedia.org/T238400 (10Ottomata) Why do all of these projects feel the need to implement their own version of JSONSchema?! (눈_눈) https://gobblin.readthedocs.io/en/lates...