[04:23:39] PROBLEM - Check the last execution of monitor_refine_mediawiki_events on an-coord1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_mediawiki_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:50:23] morninggg [05:50:31] there was a disk space issue for an-coord1001 [05:50:42] we keep indefinitely logs for hive, etc.. [05:50:44] no bueno [06:15:42] (03CR) 10Elukey: [V: 03+2 C: 03+2] "> Can we add README to this dir on how do you submit this ingestion" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538916 (https://phabricator.wikimedia.org/T229682) (owner: 10Elukey) [06:44:54] !log clean up files older than 30d in /var/log/{oozie,hive} on an-coord1001 [06:44:56] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:45:09] but in theory we have log4j set to keep max 14d [06:45:11] mmmm [07:08:42] * elukey errand for ~1h [08:20:02] Hi team - I'm back home but still need to care Naé and Lino [10:15:39] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10jrobell) Hi @Nuria and all, Jerrie is full time staff with a req number and Erin is a full time contr... [10:42:21] 10Analytics, 10Operations, 10User-Elukey: setup/install krb2001/WMF6577 - https://phabricator.wikimedia.org/T233142 (10elukey) Very strange, the debian install works in setting up raids and lvm volumes, but fail when installing grub. I noticed that the host has multiple huge disks (4TB each), and they all ge... [11:06:28] 10Analytics, 10Operations, 10User-Elukey: setup/install krb2001/WMF6577 - https://phabricator.wikimedia.org/T233142 (10MoritzMuehlenhoff) >>! In T233142#5529100, @elukey wrote: > Very strange, the debian install works in setting up raids and lvm volumes, but fail when installing grub. I noticed that the host... [11:07:44] 10Analytics, 10Operations, 10User-Elukey: setup/install krb2001/WMF6577 - https://phabricator.wikimedia.org/T233142 (10elukey) >>! In T233142#5529150, @MoritzMuehlenhoff wrote: >>>! In T233142#5529100, @elukey wrote: >> Very strange, the debian install works in setting up raids and lvm volumes, but fail when... [11:12:35] 10Analytics, 10Operations, 10User-Elukey: setup/install krb2001/WMF6577 - https://phabricator.wikimedia.org/T233142 (10elukey) The new recipe seems to have worked! [12:48:40] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: drop CitatitionUsage data on mysql - https://phabricator.wikimedia.org/T233893 (10Miriam) Hi @Nuria and @elukey - please feel free to drop this dataset from mysql. @tizianopiccardi -the main user for this data - also confirmed. [13:05:21] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: drop CitatitionUsage data on mysql - https://phabricator.wikimedia.org/T233893 (10elukey) db1107 ` MariaDB [(none)]> show tables from log like 'CitationUsage%'; +--------------------------------+ | Tables_in_log (CitationUsage%) | +------------------... [13:11:20] 10Analytics, 10Operations, 10User-Elukey: setup/install krb2001/WMF6577 - https://phabricator.wikimedia.org/T233142 (10elukey) ` elukey@krb2001:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G... [13:19:51] 10Analytics, 10Operations, 10Patch-For-Review, 10User-Elukey: setup/install krb2001/WMF6577 - https://phabricator.wikimedia.org/T233142 (10elukey) 05Open→03Resolved [13:19:54] 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: codfw: 1 misc node for the Kerberos KDC service - https://phabricator.wikimedia.org/T227425 (10elukey) [13:28:12] ok krb2001 is up and running in codfw [13:28:49] painful but now the next step is to enable/test replication between hosts [14:17:42] 10Analytics, 10Analytics-Kanban, 10Operations, 10netops, 10ops-eqiad: Move cloudvirtan* hardware out of CloudVPS back into production Analytics VLAN. - https://phabricator.wikimedia.org/T225128 (10elukey) 05Open→03Resolved ` elukey@asw2-a-eqiad> show ethernet-switching interface xe-4/0/37 Routing Ins... [14:17:45] 10Analytics, 10Cloud-Services: Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users - https://phabricator.wikimedia.org/T204950 (10elukey) [14:18:45] finally an-presto nodes ready! [14:18:50] ottomata: --^ [14:18:53] \o/ [14:21:09] what's a recommended way to move data from hadoop to stat1006 [14:21:11] ? [14:28:35] groceryheist: hi! [14:28:45] it depends how much data you want to copy :D [14:29:02] in general we discourage to move ton of GBs to the stat nodes [14:29:25] but the hdfs dfs util has copyToLocal IIRC [14:37:01] elukey: woohhoo [14:37:08] well [14:37:12] stat1006 doesn't have an hdfs client [14:37:16] groceryheist: why do you want it on stat1006? [14:38:21] ah yes forgot about it [14:38:30] I assumed he meant a generic stat bo [14:38:32] *box [14:38:43] of course stat1006 is the only one without hdfs access :D [14:57:33] 10Analytics, 10EventBus, 10Scoring-platform-team, 10Patch-For-Review: Change event.mediawiki_revision_score schema to use map types - https://phabricator.wikimedia.org/T225211 (10Ottomata) This is looking great so far! I will fix the Hive table on Monday. [15:14:01] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10EventBus, and 2 others: Figure out how to $ref common schema across schema repositories - https://phabricator.wikimedia.org/T233432 (10Ottomata) Ok, submodules it is. Next question: Should we create a new 'common wikimedia' schema repository an... [15:15:56] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: High volume mediawiki analytics events camus import is lagging - https://phabricator.wikimedia.org/T233718 (10Ottomata) Looks like all events were replayed succesfully! Phew! Am re-refining stuff now. [15:16:32] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: High volume mediawiki analytics events camus import is lagging - https://phabricator.wikimedia.org/T233718 (10Ottomata) p:05Unbreak!→03High [15:35:13] ottomata: nice --^ [15:35:26] so we don't know why camus did that right? [15:44:17] elukey: no [15:44:25] i could not figure out what was going on [15:44:25] but [15:44:39] it might have something to do with the old kafka client camus uses [15:44:55] the iterator that was empty was being returned by kafka client code, not camus code [15:45:29] and from the way the camus code was written (with e.g. System.out.println still there), it looks like the code path it was hitting wasn't supposed to happen [15:46:04] ah snap [15:46:39] is there an alternative to camus that we can explore while waiting for kafka connectors? [15:46:42] like goblin [15:46:56] gobblin sorry [15:47:25] https://gobblin.apache.org ---> awesome animation [15:47:26] haahahah [15:47:41] it is in the apache incubator [15:47:52] IIRC it was only published by linkedin right? [16:01:10] ottomata, I'm doing some crazy stuff with ORES models and I found that the dependencies I needed weren't on stat1007. [16:02:37] ping milimetric , coming to stand up? [17:02:17] * elukey off! o/ [17:41:20] a-team: is SWAP being worked on or upgraded at the moment? I just lost access to Jupyter [17:42:59] Nettrom: not that i know, let me see ops channel [17:44:07] Nettrom: no, no issues [17:44:33] hm, I can ssh to notebook1004 just fine, but both JupyterLab and the old version (/tree) returns "ERR_EMPTY_RESPONSE" [17:48:22] …and I've got access again [17:48:35] not sure if any magic was needed, but it works [17:51:03] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10Nuria) @jrobell both need phabricator accounts and ldap accounts (via creating a user in wikitech) o... [17:52:36] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban: drop CitatitionUsage data on mysql - https://phabricator.wikimedia.org/T233893 (10Nuria) +1 to droppinmg [17:53:03] 10Analytics, 10Analytics-EventLogging, 10Analytics-Kanban, 10Performance-Team: Drop Navigationtiming data entirely from mysql storage? - https://phabricator.wikimedia.org/T233891 (10Nuria) ping @Gilles to confirm this data can be dropped from mysql [18:19:18] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10DStrine) Their phab accounts are @EYener and @jkumalah I have a wiktech account but I have forgotten... [18:22:45] 10Analytics, 10Fundraising-Backlog, 10Operations, 10SRE-Access-Requests: Banner History and page view data access for fundraising analysts - Jerrie and Erin - https://phabricator.wikimedia.org/T233636 (10Nuria) @DStrine : just creating a user/password on https://wikitech.wikimedia.org/wiki/Main_Page is enough [18:25:41] (03CR) 10Nuria: "Sounds great, thanks for docs." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/538916 (https://phabricator.wikimedia.org/T229682) (owner: 10Elukey)