[00:27:44] 10Analytics, 10SRE, 10observability: Set up cross DC topic mirroring for Kafka logging clusters - https://phabricator.wikimedia.org/T276972 (10crusnov) p:05Triage→03Medium [06:17:05] !log reimage an-worker1111 to buster [06:17:08] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:17:11] good morning [06:17:15] last worker to reimage :) [06:23:56] PROBLEM - Check the last execution of refine_eventlogging_legacy on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refine_eventlogging_legacy https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:28:23] !log force the re-run of refine_eventlogging_legacy - failed due to worker reimage in progress [06:28:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:28:40] 10Analytics-Clusters, 10Patch-For-Review: Install Debian Buster on Hadoop - https://phabricator.wikimedia.org/T231067 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['an-worker1111.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/20... [06:35:10] RECOVERY - Check the last execution of refine_eventlogging_legacy on an-launcher1002 is OK: OK: Status of the systemd unit refine_eventlogging_legacy https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:38:58] 10Analytics, 10ops-eqiad: analytics1066's BBU might need to be replaced - https://phabricator.wikimedia.org/T277005 (10elukey) [06:43:06] 10Analytics, 10ops-eqiad: analytics1066's BBU might need to be replaced - https://phabricator.wikimedia.org/T277005 (10elukey) @razzi the error in icinga is `CRITICAL: 12 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteTh... [07:02:58] 10Analytics-Clusters, 10Patch-For-Review: Install Debian Buster on Hadoop - https://phabricator.wikimedia.org/T231067 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-worker1111.eqiad.wmnet'] ` and were **ALL** successful. [07:04:59] all hadoop worker nodes on buster \o/ [07:05:21] !log all hadoop worker nodes on Buster [07:05:25] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:10:57] 10Analytics-Clusters, 10Patch-For-Review: Install Debian Buster on Hadoop - https://phabricator.wikimedia.org/T231067 (10elukey) ` elukey@cumin1001:~$ sudo cumin 'A:hadoop-worker' 'cat /etc/debian_version' 78 hosts will be targeted: an-worker[1078-1128,1130-1132,1135-1138].eqiad.wmnet,analytics[1058-1077].eqia... [07:58:41] ok also updated some admin docs [08:29:28] Heya - not better today, I won't be able to put in a lot of time, and previsions for this end of week are not better :( [08:42:05] 10Analytics, 10Analytics-Kanban: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10elukey) [08:42:28] joal: take care! Don't worry about work :) [08:48:43] 10Analytics, 10Analytics-Kanban: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10elukey) I don't have permits to do: ` ssh elukey@gerrit.wikimedia.org -p 29418 'gerrit create-project -d "Package Apache Airflow" operations/debs/airflow -o ldap/ops -p operations/debs' ` [09:01:14] 10Analytics, 10Analytics-Kanban, 10Packaging: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10Peachey88) [09:07:56] 10Analytics-Clusters, 10Data-Persistence-Backup: Evaluate the need to generate and maintain zookeeper backups - https://phabricator.wikimedia.org/T274808 (10jcrespo) p:05Triage→03Low I will reuse this ticket as the implementation one, but with low priority for now. [09:09:01] 10Analytics, 10Analytics-Kanban, 10Packaging: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10elukey) https://gerrit.wikimedia.org/r/admin/repos/operations/debs/airflow (Thanks to Joe!) [09:10:58] 10Analytics-Clusters, 10Data-Persistence-Backup: Implement production zookeeper backups - https://phabricator.wikimedia.org/T274808 (10jcrespo) [09:14:37] (03CR) 10Matthias Mullie: "This patch got approved (by humans) and verified (by CI), but hasn't yet been merged/submitted." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/663703 (https://phabricator.wikimedia.org/T263154) (owner: 10Eric Gardner) [09:14:55] (03CR) 10Matthias Mullie: [C: 03+1] Update schema to 1.3.0 and add new "image" mediatype option [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668748 (owner: 10Eric Gardner) [09:20:03] 10Analytics, 10Data-Persistence-Backup: Evaluate possible solutions to backup Analytics Hadoop's HDFS data - https://phabricator.wikimedia.org/T277015 (10elukey) [09:21:35] 10Analytics, 10Data-Persistence-Backup: Evaluate possible solutions to backup Analytics Hadoop's HDFS data - https://phabricator.wikimedia.org/T277015 (10elukey) [10:21:00] 10Analytics-Clusters, 10DBA, 10Patch-For-Review: Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10Marostegui) s5 and s8 are now up and replicating [10:36:01] 10Analytics, 10Data-Persistence-Backup: Evaluate possible solutions to backup Analytics Hadoop's HDFS data - https://phabricator.wikimedia.org/T277015 (10LSobanski) @elukey thanks for reaching out, a few questions: - Is the expectations to do backups continuously or at fixed points in time? - Is the cluster in... [11:47:40] 10Analytics, 10Data-Persistence-Backup: Evaluate possible solutions to backup Analytics Hadoop's HDFS data - https://phabricator.wikimedia.org/T277015 (10elukey) Adding my thoughts about it, then my team will be able to comment :) >>! In T277015#6899852, @LSobanski wrote: > @elukey thanks for reaching out, a... [11:49:45] 10Analytics-Clusters, 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10Marostegui) All the sections have been started and are now in sync with their masters. I have run a check... [11:51:35] 10Analytics-Clusters, 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10elukey) @razzi can you follow up with @Bstorm about the next steps? :) [11:59:42] 10Analytics, 10Data-Persistence-Backup: Evaluate possible solutions to backup Analytics Hadoop's HDFS data - https://phabricator.wikimedia.org/T277015 (10jcrespo) > we don't have particular requirements for the location The answer to that would mostly be motivated by: how much time could you wait for the reco... [12:11:02] 10Analytics, 10Data-Persistence-Backup: Evaluate possible solutions to backup Analytics Hadoop's HDFS data - https://phabricator.wikimedia.org/T277015 (10elukey) @jcrespo thanks for the infos, lemme add more notes: * A day was a random value that picked turned out to be very wrong, I think that we can wait ev... [12:28:09] 10Analytics, 10Data-Persistence-Backup: Evaluate possible solutions to backup Analytics Hadoop's HDFS data - https://phabricator.wikimedia.org/T277015 (10jcrespo) > Practically this might be a little problematic in a data recovery scenario Yes, this is something that I expected, as we had a similar kind of de... [12:33:13] * elukey lunch! [13:15:11] 10Analytics-EventLogging, 10Analytics-Radar, 10Front-end-Standards-Group, 10MediaWiki-extensions-WikimediaEvents, and 4 others: Provide a reusable getEditCountBucket function for analytics purposes - https://phabricator.wikimedia.org/T210106 (10phuedx) >>! In T210106#6895285, @awight wrote: > I would love... [13:52:44] (03PS1) 10Sahilgrewalhere: Fixed typo "paramaters" [analytics/aggregator] - 10https://gerrit.wikimedia.org/r/670471 (https://phabricator.wikimedia.org/T201491) [14:01:20] (03PS1) 10Sahilgrewalhere: Fixed typo "paramaters" [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/670474 (https://phabricator.wikimedia.org/T201491) [14:10:01] (03PS1) 10Sahilgrewalhere: Fixed typo "paramaters" [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/670478 (https://phabricator.wikimedia.org/T201491) [14:12:20] (03PS6) 10Phuedx: universalLanguageSelector: Add new properties [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668743 (https://phabricator.wikimedia.org/T275766) [14:16:17] (03CR) 10jerkins-bot: [V: 04-1] universalLanguageSelector: Add new properties [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668743 (https://phabricator.wikimedia.org/T275766) (owner: 10Phuedx) [14:18:13] (03CR) 10Mholloway: "Yeah, it's time to fix this, let me look into it." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/663703 (https://phabricator.wikimedia.org/T263154) (owner: 10Eric Gardner) [14:26:04] 10Analytics, 10Event-Platform, 10Inuka-Team (Kanban): KaiOSAppFeedback Event Platform Migration - https://phabricator.wikimedia.org/T267345 (10SBisson) Let's start with this one so we figure out what to do on the "least critical" schema. [14:33:46] 10Analytics-Radar: Presto error in Superest - only when grouping - https://phabricator.wikimedia.org/T270503 (10JAllemandou) 05Open→03Resolved [14:42:21] PROBLEM - Check the last execution of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:43:29] Hi, I'm Pablo, new research scientist in the research team. I just created a table in Hive called pablo.toy and checked that I can explore it in the Superset SQL Lab. However, I am not allowed to add the table in Superset to then create graphs... how could I do this in order to create dashboards? (I imagine this action requires specific permissions) [14:43:30] https://apache-superset.readthedocs.io/en/0.28.1/security.html [14:44:47] 10Analytics, 10Analytics-Kanban, 10Packaging: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10elukey) After some chats with Riccardo and Moritz about how to package a Python app with dependencies not in Debian upstream, I ended up discovering that there is no clear w... [14:45:20] elaragon: hi! [14:45:57] can you add more details about the "not allowed" part? More specifically, what happens [14:46:03] or what error you get [14:47:24] I get a red alert message with the text: 'table' [14:48:53] elaragon: ah that is definitely weird, maybe a superset bug, can you tell me exactly how to repro? [14:49:18] (I wouldn't expect 'table' as error message :D) [14:50:23] also, does it happen with a specific chart or with all? [14:53:05] 10Analytics: Check home/HDFS leftovers of dedcode - https://phabricator.wikimedia.org/T276748 (10elukey) ` ====== stat1004 ====== total 0 ====== stat1005 ====== total 266892 -rw-r--r-- 1 22235 wikidev 4588519 Feb 12 2020 core_stable.tar.gz drwxrwxr-x 7 22235 wikidev 4096 Feb 25 2020 data -rw-r--r--... [14:53:45] RECOVERY - Check the last execution of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:54:50] 1. Access to https://superset.wikimedia.org/tablemodelview/add (login required) [14:54:53] 2. Import a table definition: Database->presto_analytics_hive and Table Name->pablo.toy [14:54:54] 3. Save [14:55:02] 10Analytics: Check home/HDFS data of Bernd Sitzmann - https://phabricator.wikimedia.org/T273712 (10elukey) ` ====== stat1004 ====== total 0 ====== stat1005 ====== total 0 ====== stat1006 ====== total 24 -rw-r--r-- 1 5037 wikidev 430 Sep 3 2014 blocked_avgs.sh -rw-r--r-- 1 5037 wikidev 230 Aug 28 2014 blocke... [14:55:24] 10Analytics, 10Analytics-Kanban, 10Packaging: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10Ottomata) Depend on anaconda-wmf? :) [14:55:34] 10Analytics: Check home/HDFS leftovers of dedcode - https://phabricator.wikimedia.org/T276748 (10elukey) p:05Triage→03Medium [14:55:36] 10Analytics: Check home/HDFS data of Bernd Sitzmann - https://phabricator.wikimedia.org/T273712 (10elukey) p:05Triage→03Medium [14:57:19] ottomata: o/ there are also kerberos stuff to add for airflow, are they all into anaconda-wmf? [14:57:29] sasl libs etc.. [14:57:54] Ok, problem solved with: Database->presto_analytics_hive, Schema->pablo; Table Name->toy [14:57:57] thanks! [14:58:07] elaragon: ah perfect! [14:58:28] elukey: they could be! [14:58:40] we could even just add airflow to anaconda-wmf if we wanted [14:58:59] ...if it is just a pip or conda package [14:58:59] ah this is interesting [14:59:05] it is a pip package yes [14:59:27] but then every time we'd need an upgrade we should upgrade anaconda as well [14:59:33] true [14:59:40] which is fine but maybe not the best [14:59:50] I mean I am fine with all the options, one less package is surely good :D [15:00:15] elukey: a nice thing about the conda stuff is that we don't have to rely on system python anymore...whiich seems to keep changing versionso through os upgrades [15:01:47] ottomata: it has pros and cons, like not getting Debian's security patches for the interpreter and the stdlib [15:03:47] (I mean it is ok but we should let Moritz know so we get pings if/when we need to upgrade anaconda-wmf) [15:06:09] ottomata: so for the conda/airflow duo, the idea would be to instruct debian rules to create a conda env and pip install airflow, only getting the extra deps needed? [15:06:12] and then deploy it [15:06:29] (not sure if it works like that or not) [15:07:10] basically, IIUC, pip install without using --ignore-installed so the local deps would be picked [15:07:31] then this "frozen" conda env will be added to the deb package [15:07:53] HMMMM that could be a way to do it [15:07:59] i hadn't thoguht of that but...yeah [15:08:11] you could just make an airflow-conda env that is totally separate from anaconda-wmf [15:08:34] following the same process that we do for anaconda-wmf..but using just conda + whatever deps you'd need [15:08:44] super ignorant about it [15:09:01] this isn't that different than using wheels + scap + pip install local venv i guess [15:09:09] it is interesting indeed [15:09:15] except it is a deb package [15:09:21] elukey: if you likek i can explain more in bc? [15:09:31] ottomata: if you have time yes! I am a total n00b [15:10:37] 10Analytics, 10Event-Platform, 10Continuous-Integration-Config: Jenkins-bot does not submit changes on passing gate-and-submit for /schemas/event/* repos - https://phabricator.wikimedia.org/T277051 (10Mholloway) [15:10:56] (03CR) 10Mholloway: "> Patch Set 4:" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/663703 (https://phabricator.wikimedia.org/T263154) (owner: 10Eric Gardner) [15:14:35] 10Analytics, 10Event-Platform, 10Continuous-Integration-Config: Jenkins-bot does not submit changes on passing gate-and-submit for /schemas/event/* repos - https://phabricator.wikimedia.org/T277051 (10hashar) a:03hashar Sounds like Gerrit permissions issues. The `integration` group should be granted the `S... [15:17:46] 10Analytics, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: [Metrics Platform] Define stream configuration syntax relevant to v1 release - https://phabricator.wikimedia.org/T273235 (10jlinehan) a:03jlinehan [15:22:50] 10Analytics, 10Event-Platform, 10Continuous-Integration-Config, 10Patch-For-Review: Jenkins-bot does not submit changes on passing gate-and-submit for /schemas/event/* repos - https://phabricator.wikimedia.org/T277051 (10hashar) I have created two new repositories for permissions purposes: * `schema` * `sc... [15:23:47] 10Analytics, 10Event-Platform, 10Continuous-Integration-Config, 10Patch-For-Review: Jenkins-bot does not submit changes on passing gate-and-submit for /schemas/event/* repos - https://phabricator.wikimedia.org/T277051 (10hashar) 05Open→03Resolved Should be good now. Please reopen if that still fails! [15:29:05] ottomata: hey, when you have a minute can you take a look at https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/670288 ? I was going to deploy to staging and canary first and likely also send some test events myself [15:55:41] (03CR) 10Phuedx: [C: 04-1] "See inline." (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668743 (https://phabricator.wikimedia.org/T275766) (owner: 10Phuedx) [16:17:40] 10Analytics, 10Analytics-Kanban, 10Packaging: Create a debian package for Apache Airflow - https://phabricator.wikimedia.org/T277012 (10elukey) Had a chat with Andrew over meet, I'll try to come up with something similar to `anaconda-wmf`, since it is a lot of good things that might be useful. The idea would... [16:23:34] 10Analytics, 10SRE, 10observability: Set up cross DC topic mirroring for Kafka logging clusters - https://phabricator.wikimedia.org/T276972 (10Ottomata) Basically, a non-aggregate Kafka cluster (like Kafka jumbo) is the source of stream data. Here, a 'stream' refers to mulitple topics, in our case, every DC... [16:28:29] 10Analytics, 10Machine-Learning-Team: Configure the Hadoop cluster to use the GPUs available on some workers - https://phabricator.wikimedia.org/T276791 (10elukey) [16:28:37] 10Analytics-Clusters: Configure Yarn to be able to locate nodes with a GPU - https://phabricator.wikimedia.org/T264401 (10elukey) 05Open→03Stalled This is stalled until we get to the Capacity scheduler :) [16:29:25] 10Analytics, 10Machine-Learning-Team: Configure the Hadoop cluster to use the GPUs available on some workers - https://phabricator.wikimedia.org/T276791 (10elukey) After some thoughts it feels better in my opinion to see if we can switch to the Yarn capacity scheduler, and then apply labels as we originally th... [16:36:45] 10Analytics: Review the Yarn Capacity scheduler and see if we can move to it - https://phabricator.wikimedia.org/T277062 (10elukey) [16:40:42] (03CR) 10Mholloway: [C: 03+2] Update schema to 1.3.0 and add new "image" mediatype option [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668748 (owner: 10Eric Gardner) [16:41:27] 10Analytics: Review the Yarn Capacity scheduler and see if we can move to it - https://phabricator.wikimedia.org/T277062 (10elukey) [16:42:11] (03Merged) 10jenkins-bot: Update schema to 1.3.0 and add new "image" mediatype option [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668748 (owner: 10Eric Gardner) [16:45:44] 10Analytics, 10Event-Platform, 10Continuous-Integration-Config: Jenkins-bot does not submit changes on passing gate-and-submit for /schemas/event/* repos - https://phabricator.wikimedia.org/T277051 (10Mholloway) Looking good! Thank you, @hashar! [16:45:47] (03CR) 10Ottomata: "Nit on naming." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan) [16:46:14] (03CR) 10Ottomata: "(Weird sorry, not sure why that just posted again, had a tab open with a draft I think)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668244 (owner: 10Sharvaniharan) [16:51:49] 10Analytics-Radar, 10Cassandra, 10ContentTranslation, 10Event-Platform, and 9 others: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10hnowlan) [16:53:41] 10Analytics-Clusters, 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10razzi) Sounds good @elukey. Thanks for your speedy data population @Marostegui! Responding to the firewa... [16:54:45] 10Analytics-Clusters, 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Convert labsdb1012 from multi-source to multi-instance - https://phabricator.wikimedia.org/T269211 (10Marostegui) @razzi yeah, it was all fixed by removing the DNS IPv6 record. Nothing else required. [16:54:57] !log rebalance kafka partitions for webrequest_upload partition 15 [16:54:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:56:48] razzi: did we reboot matomo1002? If not let's find some time [16:57:00] elukey: have not yet, can do today! [16:57:01] also an-conf100[1-3] need to be rebooted [17:12:39] 10Analytics-Radar, 10Cassandra, 10ContentTranslation, 10Event-Platform, and 10 others: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10WDoranWMF) [17:16:13] (03CR) 10DannyS712: [C: 03+1] Fixed typo "paramaters" [analytics/pivot/deploy] - 10https://gerrit.wikimedia.org/r/670478 (https://phabricator.wikimedia.org/T201491) (owner: 10Sahilgrewalhere) [17:16:56] (03CR) 10DannyS712: [C: 03+1] Fixed typo "paramaters" [analytics/aggregator] - 10https://gerrit.wikimedia.org/r/670471 (https://phabricator.wikimedia.org/T201491) (owner: 10Sahilgrewalhere) [17:19:47] (03CR) 10jerkins-bot: [V: 04-1] Fixed typo "paramaters" [analytics/aggregator] - 10https://gerrit.wikimedia.org/r/670471 (https://phabricator.wikimedia.org/T201491) (owner: 10Sahilgrewalhere) [17:23:51] DISBAND!!! [17:23:57] that's the word I was looking for [17:24:08] :] [17:24:44] ottomata: is this enough for switching session_tick to 100% sampling on testwiki? https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/670509 [17:34:17] mforns: they are looking at that too! [17:34:18] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/670496 [17:34:22] come to mep sync [17:43:26] * razzi lunchtime [18:04:01] 10Analytics: Check home/HDFS leftovers of dedcode - https://phabricator.wikimedia.org/T276748 (10MGerlach) @elukey thanks for the ping. I just talked with Djellel. - hdfs/hive: all data can be dropped - stat100X[5,6,7,8]: /user/dedcode/: is this possible to keep for some time? we are mainly interested in keepi... [18:15:20] elukey: I didn't follow up with you yesterday on the session length deployment... sorry. In the end, there will be no changes, so we can deploy as is. If it's OK for you, I'll deploy now, and try to start the job before my sync-up with buod. [18:16:52] !log starting deployment of refinery (session length oozie job) [18:17:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:24:18] 10Analytics, 10Product-Infrastructure-Team-Backlog, 10Wikimedia Taiwan, 10Chinese-Sites, 10Pageviews-Anomaly: Top read is showing one page that had fake traffic in zhwiki - https://phabricator.wikimedia.org/T274605 (10MSantos) @Htchien please reach me at msantos@wikimedia.org [18:33:42] mforns: I wanted to ask you if it was ok to deploy earlier on and I forgot (I am in a meeting now), +1 from me for the deployment [18:33:48] sorry that you have to do it :( [18:34:06] elukey: not at all! :] [18:34:40] actually I'm the 2nd ops weeker this week, so.. [18:40:32] mforns: <3 [18:44:40] !log finished deployment of refinery (session length oozie job) [18:44:42] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:53:48] 10Analytics: Check home/HDFS leftovers of dedcode - https://phabricator.wikimedia.org/T276748 (10elukey) @MGerlach I can move the /home/dedcode dirs under your username, what we care is that an active user maintains/own them so we can ping in case there are issues etc... Would it be ok? Then you'll be in charge... [19:13:40] mforns: going afk in a bit, anything that I can help with? [19:13:51] elukey: no no, all good, thanks! [19:13:55] thank you! [19:13:57] * elukey afk! [19:14:08] 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure: Roll-up raw sessionTick data into distribution - https://phabricator.wikimedia.org/T271455 (10kzimmerman) a:05Mayakp.wiki→03kzimmerman [20:03:19] 10Analytics, 10DC-Ops, 10SRE, 10ops-eqiad: analytics1066's BBU might need to be replaced - https://phabricator.wikimedia.org/T277005 (10crusnov) p:05Triage→03Medium [20:03:42] ottomata: hey uh I don't wanna nag, but, nagging you again about https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/670288 :) [20:04:25] +11 [20:04:27] ! [20:04:36] smoehow missed that cdanis thank you for nag! [20:04:37] thanks! [20:04:52] my plan is to push to staging and then canary and make sure events are getting annotated properly [20:07:48] great [20:32:05] (03CR) 10Krinkle: universalLanguageSelector: Add new properties (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/668743 (https://phabricator.wikimedia.org/T275766) (owner: 10Phuedx) [20:44:14] ottomata: have a second for me and this Kafka consumer? [20:51:14] milimetric: i have 9 minutes, or more time in 45 minutes! [20:51:51] ottomata: um, after your meeting is ok, shouldn't take long but no need to rush it [20:52:20] k! [21:28:45] milimetric: [21:28:48] bc? [21:28:59] yeah, omw [21:36:10] 10Analytics, 10SRE, 10Patch-For-Review: Augment NEL reports with GeoIP country code and network AS number - https://phabricator.wikimedia.org/T263496 (10CDanis) 05Open→03Resolved ASN, ISP/organization, country, & subdivision are now visible in Logstash! [22:19:22] 10Analytics, 10FR-Tech-Analytics, 10Fundraising-Backlog: Whitelist Portal and WikipediaApp event data for (sanitized) long-term storage - https://phabricator.wikimedia.org/T273246 (10EYener) Hi @mforns getting back to you on this. I'll schedule a meeting for next week with you, myself, @Jdrewniak, @mpopov ,... [23:15:04] !log rebalance kafka partitions for webrequest_upload partition 16 [23:15:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log