[04:14:12] <wikibugs>	 (03PS9) 10Razzi: Upgrade superset to 1.0.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/665130 (https://phabricator.wikimedia.org/T272390)
[04:53:21] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10fkaelin) After going a little overboard still no easy answers. I did slice and dice the data based on the query that @Milimetric provided abov...
[05:11:13] <wikibugs>	 (03PS10) 10Razzi: Upgrade superset to 1.0.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/665130 (https://phabricator.wikimedia.org/T272390)
[05:34:09] <wikibugs>	 (03PS11) 10Razzi: Upgrade superset to 1.0.1 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/665130 (https://phabricator.wikimedia.org/T272390)
[05:40:48] <razzi>	 !log sudo chown analytics /var/log/refinery/sqoop-mediawiki.log.1 on an-launcher1002 and restart logrotate
[05:40:52] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[05:41:39] <razzi>	 There was a permission error on that one file, strangely, and logrotate had exited
[05:41:57] <razzi>	 Probably some sqoop job was run / is running as root and shouldn't be
[05:49:31] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10razzi) Alright, building the superset wheel from the latest upstream source on our build server deneb.codfw.wmnet produced a wheel that is 25M, which is in line with the size o...
[06:51:48] <elukey>	 good morning
[06:52:18] <joal>	 good morning
[08:12:55] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "Keith, do you plan to use this on the new Buster mwlog hosts? If so, I can merge and rebuild the deb beforehand (or we roll this one out l" [analytics/udplog] - 10https://gerrit.wikimedia.org/r/673596 (https://phabricator.wikimedia.org/T276623) (owner: 10Majavah)
[08:32:08] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bdijkstra) I have some Quarry-queries that use information from multiple databases. Will these continue to work?
[09:34:41] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bdijkstra) When I modify one of my existing queries and click 'Submit', nothing happens. When after a few minutes I press F5, I see that the changes have not been saved. In th...
[09:44:02] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10CommanderWaterford) No matter which DB you enter into the DB Name Field - nothing happens, query never gets executed. Absolutely no-go right now!
[10:50:23] <wikibugs>	 10Analytics-Radar, 10Cassandra, 10ContentTranslation, 10Event-Platform, and 9 others: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10MSantos)
[10:53:28] <elukey>	 joal: bonjour bonjour
[10:53:35] <elukey>	 time to chat about alluxio?
[10:53:42] <elukey>	 of course not, it is not today
[10:53:53] <elukey>	 sorryyyy
[11:21:09] * elukey lunch!
[12:01:43] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10UOzurumba) Hello @Bstorm,   Please, I am having issues with Quarry; when I try to fork an existing quarry and click on "submit query", it pops up a message: "quarry.wmflabs.or...
[12:44:41] <wikibugs>	 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure, and 2 others: prefUpdate schema contains multiple identical events for the same preference update - https://phabricator.wikimedia.org/T218835 (10Edtadros) === Test Result - Beta  **Status:** ✅ PASS **Environment:**...
[12:45:21] <wikibugs>	 10Analytics-Radar, 10Better Use Of Data, 10Product-Analytics, 10Product-Data-Infrastructure, and 2 others: prefUpdate schema contains multiple identical events for the same preference update - https://phabricator.wikimedia.org/T218835 (10Edtadros)
[13:02:29] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10Ottomata) Nice!
[13:50:47] <wikibugs>	 10Analytics-Radar, 10Cassandra, 10ContentTranslation, 10Event-Platform, and 9 others: Rebuild all blubber build docker images running on kubernetes - https://phabricator.wikimedia.org/T274262 (10Ottomata)
[13:53:30] <wikibugs>	 10Analytics, 10Product-Analytics: Hive Runtime Error - Query on event.MobileWikiAppDailyStats failing with errors - https://phabricator.wikimedia.org/T277348 (10Ottomata) Hiya, just FYI reading this task, it looks like the offending error is  ` Caused by: java.lang.ClassCastException: org.apache.hadoop.hive.se...
[14:23:32] <fkaelin>	 hi. is information about tags https://en.wikipedia.org/wiki/Special:Tags available in analytics revision data?
[14:30:22] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Wikidata Usage and Coverage ETL failing from stat1004 - https://phabricator.wikimedia.org/T278299 (10Addshore)
[14:39:16] <wikibugs>	 10Analytics, 10DBA, 10Event-Platform, 10WMF-Architecture-Team, 10Services (later): Consistent MediaWiki state change events | MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242 (10Ottomata)
[14:40:04] <wikibugs>	 10Analytics, 10DBA, 10Event-Platform, 10WMF-Architecture-Team, 10Services (later): Consistent MediaWiki state change events | MediaWiki events as source of truth - https://phabricator.wikimedia.org/T120242 (10Ottomata) > I'll try to get together a concrete list of services that currently do rely on Media...
[15:02:15] <milimetric>	 fkaelin: yes, revision_tags on https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/MediaWiki_history#Schema
[15:04:12] <milimetric>	 but surprisingly not https://github.com/wikimedia/schemas-event-primary/blob/master/jsonschema/mediawiki/revision/create/current.yaml
[15:04:41] <milimetric>	 oh, fkaelin: https://github.com/wikimedia/schemas-event-primary/blob/60aab4bdc3aecc41034bc966a05bd8a3e593427d/jsonschema/mediawiki/revision/tags-change/current.yaml
[15:07:37] <Nettrom>	 fkaelin: one thing to note about revision tags is that we now have revert tags that can get applied post-edit, but the time difference isn't reflected in mediawiki history
[15:09:28] <Nettrom>	 mediawiki_revision_tags_change does track that though, the timestamp of the event and the revision will reflect that difference
[15:46:28] <razzi>	 !log rebalance kafka partitions for webrequest_text partitions 3 and 4
[15:46:32] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:58:32] <wikibugs>	 10Analytics, 10WMDE-Analytics-Engineering, 10User-GoranSMilovanovic: Wikidata Usage and Coverage ETL failing from stat1004 - https://phabricator.wikimedia.org/T278299 (10GoranSMilovanovic) p:05High→03Medium - The system was migrated to `stat1008`; - No problems observed during a manual, test update; - Cr...
[16:40:34] <wikibugs>	 10Quarry: Quarry queries forever stuck in queue - https://phabricator.wikimedia.org/T274071 (10Bdijkstra) I think it's happening again. Some queries have been the 'running' state for more than 12 hours.
[16:41:17] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10fkaelin) Thanks for the background on where the special:tag info is stored @Milimetric on irc, though since it is in the refined revisions and...
[16:51:46] <wikibugs>	 10Analytics: Review the usage of dns_canonicalize=false for Kerberos - https://phabricator.wikimedia.org/T278353 (10elukey)
[16:53:40] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10Pchelolo) I think I know what the issue is.  Actually inserting into the revision table in MW happens in RevisionStore::insertRevisionOn. Howe...
[16:54:05] <ottomata>	 elukey: " work in progress to kerberize ssh logins."... ? oh oh?
[16:54:14] <elukey>	 yep! 
[16:54:20] <ottomata>	 ticket?
[16:54:35] <elukey>	 no idea, Moritz is working on it
[16:54:55] <elukey>	 the main goal is to allow, IIUC, cumin to run command on nodes as non-root users
[16:55:48] <elukey>	 https://phabricator.wikimedia.org/T244840
[16:55:50] <elukey>	 ottomata: --^
[16:58:15] <elukey>	 the main problem for us is python, I want to make sure that things like pyspark work fine on workers
[16:58:26] <elukey>	 the main problem will be for clients
[16:58:34] <elukey>	 (using analytics-hive.eqiad.wmnet)
[17:04:10] <elukey>	 (kerberos is a love that never ends)
[17:08:39] <joal>	 Hi team - my apologozes for the missed meetings without e-scrumn
[17:11:18] <joal>	 elukey: would ou have minute to do some small tests on capacity scheduler?
[17:15:07] <ottomata>	 elukeyaye cool makes sense ty 
[17:18:17] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10Pchelolo) Ok. Houston, we do have an implementation problem. I'm fairly certain sending the event from `RevisionRecordInserted` hook would mit...
[17:20:35] <elukey>	 joal: I am in a meeting but I can be free in ~30/40 mins
[17:20:42] <joal>	 works for me elukey :)
[17:20:45] <elukey>	 super
[17:20:46] <joal>	 thanks!
[17:21:07] <joal>	 milimetric: would you by any chance be nearby?
[17:22:43] <milimetric>	 joal: I'm at the dentist, my crown cracked :(
[17:22:50] <joal>	 Arf :(
[17:22:52] <joal>	 not cool
[17:22:55] <joal>	 good luck milimetric 
[17:23:19] <milimetric>	 it's amazing, I cracked it during lunch and immediately called, someone had cancelled now I'm waiting for the doc
[17:24:39] <Pchelolo>	 hey folks. I think I found out why revision-create are missing. Cause we're not sending them!
[17:24:58] <joal>	 Pchelolo: I saw your comment on tha task - This is awesome finding
[17:25:10] <joal>	 Pchelolo: I wonder why we have not caught that in analysis
[17:25:36] <Pchelolo>	 cause it's tricky.. the code there is rather tricky.. 
[17:26:22] <Pchelolo>	 the question now is what to do about rev_is_revert problem I described. I'll leave it up to yall to chime in, I don't really know if rev_is_revert is already used and what's up there
[17:30:22] <elukey>	 joal: ok ready
[17:30:26] <elukey>	 if you are
[17:58:11] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10razzi) 05Open→03Stalled Moving this to paused while I await feedback from Product Analytics. Only known issues are with druid datasources, which can be migrated to druid ta...
[17:58:13] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Tracking-Neverending: Superset Updates - https://phabricator.wikimedia.org/T211706 (10razzi)
[18:01:29] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10elukey) @razzi if you have tested Superset and it is good, let's add a deadline for the feedback and set an upgrade day (so we don't wait too much and we can move on)
[18:07:22] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10Ottomata) Would it be possible to get `EditResult` or whatever info is needed into RevisionRecordInserted we need?  Is rev_is_revert the only...
[18:15:35] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10Pchelolo) > Would it be possible to get EditResult or whatever info is needed into RevisionRecordInserted we need?   Possible - yes. Desirable...
[18:15:49] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Product-Data-Infrastructure: Define acceptable usage of the `meta` object in event schemas - https://phabricator.wikimedia.org/T273293 (10LGoto) p:05Triage→03Medium a:03jlinehan
[18:17:51] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi) Thanks for your comment @ema. I'll see if I can do the same with [kafka-kit](https://github.com/DataDog/kafka-kit/).
[18:18:30] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban: Balance Kafka topic partitions on Kafka Jumbo to take advantage of the new brokers - https://phabricator.wikimedia.org/T255973 (10razzi)
[18:20:45] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Growth-Team, 10Product-Analytics: Revisions missing from mediawiki_revision_create - https://phabricator.wikimedia.org/T215001 (10Ottomata) We just added rev_is_revert a [[ https://phabricator.wikimedia.org/T216297#6362930 | few months ago ]].  It really only indicated if...
[18:22:21] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Analytics, and 2 others: [MEP] [BUG] dt field in migrated client-side EventLogging schemas is not set to meta.dt - https://phabricator.wikimedia.org/T277330 (10jlinehan)
[18:24:28] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bstorm) >>! In T264254#6940389, @CommanderWaterford wrote: > No matter which DB you enter into the DB Name Field - nothing happens, query never gets executed. Absolutely no-go...
[18:24:49] <milimetric>	 joal: ok, back if you're still around
[18:25:01] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Analytics, and 2 others: [MEP] [BUG] dt field in migrated client-side EventLogging schemas is not set to meta.dt - https://phabricator.wikimedia.org/T277330 (10LGoto) a:03kzimmerman
[18:25:37] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Data-Infrastructure, 10Product-Analytics (Kanban): [MEP] [BUG] Timestamp format changed in migrated client-side EventLogging schemas - https://phabricator.wikimedia.org/T277253 (10LGoto) a:03kzimmerman
[18:25:41] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bstorm) >>! In T264254#6940373, @Bdijkstra wrote: > When I modify one of my existing queries and click 'Submit', nothing happens. When after a few minutes I press F5, I see th...
[18:25:51] <elukey>	 razzi: how many sqoop processes/tests are running on an-launcher1002?
[18:26:12] <elukey>	 elukey@an-launcher1002:~$ uptime 18:26:04 up 29 days,  2:41,  3 users,  load average: 184.75, 184.28, 183.70
[18:27:05] <elukey>	 https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=9&orgId=1&refresh=5m&var-server=an-launcher1002&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics&from=now-2d&to=now
[18:27:17] <razzi>	 elukey: not sure, I think milimetric kicked a run off 2 days ago?
[18:27:22] <elukey>	 there was a systemd daemon flapping and causing alerts in #operations
[18:27:26] <razzi>	 uh oh
[18:27:33] <milimetric>	 hm
[18:27:44] <milimetric>	 it's just the same job that would normally run monthly
[18:28:01] <elukey>	 something must be different since we never seen a load of 180
[18:28:13] <milimetric>	 checking on the job now
[18:29:47] <milimetric>	 it's on wiki 312 of 892, going pretty fast
[18:29:50] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bstorm) >>! In T264254#6940914, @UOzurumba wrote: > Hello @Bstorm,   > Please, I am having issues with Quarry; when I try to fork an existing quarry and click on "submit query...
[18:30:33] <elukey>	 mmm I see the timedatad daemon (part of systemd, the one alarmin) with a lot of D processes, in uninterruptible sleep, maybe it is related
[18:30:36] <milimetric>	 no errors or warnings, so the job's running as expected
[18:30:56] <milimetric>	 what are D processes?
[18:31:35] <elukey>	 uninterruptible sleep
[18:31:49] <elukey>	 usually they are stuck in something, can't be killed
[18:32:43] <elukey>	 milimetric: do you recall more or less when you kicked off the sqoop run?
[18:32:50] <elukey>	 does it match with
[18:32:51] <elukey>	 https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=9&orgId=1&refresh=5m&var-server=an-launcher1002&var-datasource=eqiad%20prometheus%2Fops&var-cluster=analytics&from=now-2d&to=now 
[18:32:55] <elukey>	 with the ramp up I mean
[18:32:57] <milimetric>	 I'll tell you exactly
[18:32:58] <elukey>	 ?
[18:33:12] <milimetric>	 group by wiki_db
[18:33:15] <milimetric>	 heh, no...
[18:33:22] <milimetric>	 2021-03-23T13:58:48
[18:33:58] <elukey>	 UTC?
[18:34:03] <milimetric>	 so, no, my job starts much later, yeah it's UTC
[18:35:08] <elukey>	 so the CPU usage is not affected that much, I think the load is artificial, due to all those processes stuck in D state
[18:35:13] <milimetric>	 the first test, on the three smaller wikis was a few hours earlier, so that's also not coinciding with the ramp-up.  It looks like something to do with replication?
[18:35:23] <elukey>	 every time that systemd tries to restart it fails and create another D process
[18:35:35] <milimetric>	 I see
[18:35:43] <elukey>	 it is an-launcher1002, this is the strange thing
[18:35:52] <milimetric>	 oh it's on the launcher?!
[18:35:57] <milimetric>	 sorry I'm just catching up
[18:36:15] <milimetric>	 ok, yeah, I mean, my process is just one process and it's running fine, I didn't launch it through systemd, just ran the script
[18:36:28] <milimetric>	 who do all the D processes belong to?
[18:36:32] <milimetric>	 analytics?
[18:36:56] <elukey>	 nono root, it is the timedated daemon
[18:37:05] <elukey>	 milimetric: can you stop you sqoop run?
[18:37:09] <elukey>	 to see if it recovers
[18:37:30] <elukey>	 if not we'll have to reboot
[18:37:39] <milimetric>	 sure, I feel fairly confident sqoop's going to be ok with regard to clouddb1021, this is a separate issue we can test for, right?
[18:37:44] <milimetric>	 killing
[18:37:54] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban: /wmf/data/raw should be readable by analytics-privatedata-users - https://phabricator.wikimedia.org/T275396 (10Ottomata) Checked some recently imported raw data and it is readable by analytics-privatedata-users.
[18:37:58] <elukey>	 yes yes thanks a lot :)
[18:37:59] <milimetric>	 killed
[18:38:05] * milimetric sheds a tear
[18:38:17] <elukey>	 yeah I know
[18:38:19] <wikibugs>	 10Analytics-Radar, 10Better Use Of Data, 10MediaWiki-API, 10Platform Engineering, and 2 others: Load API request count and latency data from Hadoop to a dashboard - https://phabricator.wikimedia.org/T108414 (10LGoto) p:05Triage→03High
[18:39:38] <elukey>	 ok I also see something like 
[18:39:39] <elukey>	 D    Mar23   0:00  \_ /usr/bin/lsof +c 15 -nXd DEL -e /mnt/hdfs
[18:39:44] <elukey>	 that is not good
[18:40:19] <elukey>	 they all seem to come from /usr/local/sbin/wmf-auto-restart
[18:40:50] <elukey>	 time seems to match
[18:42:01] <elukey>	 ok lemme try to force a umount of /mnt/hdfs
[18:43:14] <elukey>	 ah snap it was that
[18:43:16] * elukey sighs
[18:43:23] <elukey>	 load is dropping
[18:43:58] <elukey>	 !log kill fuse hdfs mount process on an-launcher1002, re-mounted /mnt/hdfs, too many processes in D state 
[18:44:01] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:44:23] <elukey>	 sorry milimetric will all the java processes I didn't see the real reason
[18:46:15] <elukey>	 ok so I found the issue, I'll follow up with Moritz about it
[18:46:30] <icinga-wm>	 PROBLEM - Check unit status of refinery-import-page-current-dumps on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refinery-import-page-current-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[18:46:54] <milimetric>	 elukey: maybe it was still the sqoop as it reads from the fuse mount... wait does it... no
[18:47:01] <elukey>	 nono 
[18:47:02] <milimetric>	 I donno
[18:47:21] <elukey>	 it is wmf-auto-restart, it should do lsof on /mnt/hdfs, too brittle
[18:47:33] <elukey>	 if the fuse mountpoint hangs for some reason, all is stalled
[18:47:34] <milimetric>	 ok, no worries either way, I think the test is solid, all good
[18:47:44] <elukey>	 milimetric: if you want you can kick it off again
[18:48:13] <milimetric>	 nah, too resource intensive and no reason to suspect failures for the teeny tiny wikis after hundreds of them ran fine
[18:48:36] <icinga-wm>	 PROBLEM - Check unit status of refinery-import-page-history-dumps on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit refinery-import-page-history-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[18:49:26] <elukey>	 !log systemctl restart refinery-import-* failed jobs (/mnt/hdfs errors due to me umounting the mountpoint)
[18:49:28] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:52:24] <elukey>	 razzi: this issue was a little weird, lemme know if you have douts
[18:52:34] <elukey>	 *doubts
[18:57:44] <icinga-wm>	 RECOVERY - Check unit status of refinery-import-page-current-dumps on an-launcher1002 is OK: OK: Status of the systemd unit refinery-import-page-current-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[18:59:50] <icinga-wm>	 RECOVERY - Check unit status of refinery-import-page-history-dumps on an-launcher1002 is OK: OK: Status of the systemd unit refinery-import-page-history-dumps https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[19:03:24] <wikibugs>	 10Analytics, 10SRE: wmf-auto-restart.py + lsof + /mnt/hdfs may need to tuned - https://phabricator.wikimedia.org/T278371 (10elukey)
[19:03:45] <elukey>	 milimetric,razzi opened --^
[19:03:50] <elukey>	 it contains the details of the findings
[19:04:02] <milimetric>	 cool, thanks!  Yeah seems totally unrelated
[19:04:11] <wikibugs>	 10Analytics, 10SRE: wmf-auto-restart.py + lsof + /mnt/hdfs may need to be tuned - https://phabricator.wikimedia.org/T278371 (10elukey)
[19:04:31] <elukey>	 all right logging off :)
[19:04:37] <elukey>	 have a good rest of the day folks
[19:05:01] <razzi>	 Cya elukey!
[19:05:01] <razzi>	 I'm unfamiliar with these system so I'll do some reading
[19:09:32] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10CommanderWaterford) Hi Bstorm, it is now working, no idea why, the whole morning it did not work.
[19:09:51] <joal>	 pfffff
[19:10:07] <joal>	 my irc got stuck and I missed all the pings :(
[19:10:12] <joal>	 ok let's try again
[19:10:17] <joal>	 milimetric: any chance now?
[19:10:23] <milimetric>	 yea!
[19:10:26] <milimetric>	 omw cave
[19:11:39] <milimetric>	 joal: ^
[19:11:49] <joal>	 Ack!
[19:36:09] <milimetric>	 ottomata: we have a q for you
[19:36:10] <joal>	 ottomata: Hi! would you have a minute?
[19:36:19] <joal>	 ottomata: we're in da cave with milimetric 
[19:36:30] <milimetric>	 you can come to the cave, but basically can we pull from webrequest topics with gobblin right now?
[19:36:38] <milimetric>	 (duplicating the read with camus)
[19:57:05] <wikibugs>	 10Quarry, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bdijkstra) >>! In T264254#6942792, @Bstorm wrote: >>>! In T264254#6940373, @Bdijkstra wrote: >> When I modify one of my existing queries and click 'Submit', nothing happens. W...
[19:58:17] <ottomata>	 OH
[19:58:21] <ottomata>	 joal:  just saw the ping cmoing
[20:13:15] <wikibugs>	 (03PS1) 10Bstorm: database field: if the field is NULL, don't fill it in with None [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/674697 (https://phabricator.wikimedia.org/T264254)
[20:28:35] <razzi>	 Superset update: I've asked product analytics and I'll ask here as well, please test the latest superset deployment on staging by running `ssh -NL 8080:an-tool1005.eqiad.wmnet:80 stat1004.eqiad.wmnet`. Comment on https://phabricator.wikimedia.org/T272390 if you find any issues. If everything looks good we'll deploy to superset.wikimedia.org next Wednesday. So test your dashboards before then!
[20:30:12] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10razzi) Good idea @elukey, posted in slack channel product-analytics and #wikimedia-analytics irc, and I'll share here as well: feedback is open for testing on superset staging...
[20:52:34] <wikibugs>	 10Analytics-Kanban, 10Better Use Of Data, 10Product-Analytics, 10Tracking-Neverending: Superset Updates - https://phabricator.wikimedia.org/T211706 (10razzi)
[20:52:56] <wikibugs>	 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade to Superset 1.0 - https://phabricator.wikimedia.org/T272390 (10razzi) 05Stalled→03Open Also I don't think "Stalled" is what I was looking for; I found a better action moving this to "In Code Review" on the kanban.
[21:03:10] <wikibugs>	 (03CR) 10Bstorm: "This is using this code, livehacked in and the database field has been set to NULL https://quarry-dev.wmflabs.org/query/45" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/674697 (https://phabricator.wikimedia.org/T264254) (owner: 10Bstorm)
[21:33:20] <wikibugs>	 10Analytics, 10Product-Analytics: Hive Runtime Error - Query on event.MobileWikiAppDailyStats failing with errors - https://phabricator.wikimedia.org/T277348 (10JAllemandou) Hi @SNowick_WMF  - I apologize for not having looked into your problem faster .  First a simple one: the second query (in comment) about...
[21:43:43] <wikibugs>	 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Product-Analytics, and 2 others: [MEP] [BUG] dt field in migrated client-side EventLogging schemas is not set to meta.dt - https://phabricator.wikimedia.org/T277330 (10kzimmerman) a:05kzimmerman→03nettrom_WMF Morten is going to re-run queries that...
[21:57:02] <wikibugs>	 (03PS1) 10Bstorm: worker: update some worker code for errors and connections [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/674723 (https://phabricator.wikimedia.org/T264254)
[22:08:46] <wikibugs>	 10Quarry, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bstorm) >>! In T264254#6943061, @CommanderWaterford wrote: > Hi Bstorm, it is now working, no idea why, the whole morning it did not > work.  I have a th...
[22:11:33] <wikibugs>	 10Quarry, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bstorm) >>! In T264254#6943333, @Bdijkstra wrote: >>>! In T264254#6942792, @Bstorm wrote: >>>>! In T264254#6940373, @Bdijkstra wrote: >>> When I modify o...
[22:53:11] <wikibugs>	 (03PS2) 10Bstorm: database field: if the field is NULL, don't fill it in with None [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/674697 (https://phabricator.wikimedia.org/T264254)
[22:53:43] <wikibugs>	 (03CR) 10Bstorm: "This version seems to work equally well in dev." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/674697 (https://phabricator.wikimedia.org/T264254) (owner: 10Bstorm)
[22:55:31] <wikibugs>	 10Analytics-Radar, 10Data-Persistence (Consultation), 10Platform Engineering Roadmap Decision Making, 10Epic, and 3 others: Remove revision_comment_temp and revision_actor_temp - https://phabricator.wikimedia.org/T215466 (10tstarling)
[22:55:47] <wikibugs>	 10Analytics-Radar, 10Data-Persistence (Consultation), 10Platform Engineering Roadmap Decision Making, 10Epic, and 3 others: Remove revision_comment_temp and revision_actor_temp - https://phabricator.wikimedia.org/T215466 (10tstarling)
[23:27:06] <wikibugs>	 10Quarry, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 (10Bdijkstra) >>! In T264254#6943734, @Bstorm wrote: >>>! In T264254#6943333, @Bdijkstra wrote: >>>>! In T264254#6942792, @Bstorm wrote: >>>>>! In T264254#6...