[00:30:22] <wikibugs>	 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Rerunning Spark functions with changed settings has no effect - https://phabricator.wikimedia.org/T273210 (10nshahquinn-wmf)
[01:46:44] <icinga-wm>	 PROBLEM - Host an-worker1098.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[01:46:52] <icinga-wm>	 PROBLEM - Host analytics1073.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[01:51:02] <icinga-wm>	 PROBLEM - Host an-worker1087.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[01:51:12] <icinga-wm>	 PROBLEM - Host an-worker1130.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[02:04:30] <wikibugs>	 10Analytics-Jupyter, 10Data-Engineering, 10Product-Analytics: Replace anaconda-wmf with smaller, non-stacked Conda environments - https://phabricator.wikimedia.org/T302819 (10Ottomata) Sounds fine to me!  I don't have much of an opinion, so if @aqu and @xcollazo are good with that it should be fine!  Sounds...
[02:30:55] <icinga-wm>	 RECOVERY - Host an-worker1130.mgmt is UP: PING OK - Packet loss = 0%, RTA = 0.05 ms
[04:51:43] <icinga-wm>	 PROBLEM - DNS on an-worker1130.mgmt is CRITICAL: DNS CRITICAL - expected 0.0.0.0 but got 10.65.0.156 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[04:53:32] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: Design Schema for page state and page state with content (enriched) streams - https://phabricator.wikimedia.org/T308017 (10tstarling) I think `is_temp` is fine. I don't think it's future-proof to add `user_type` since if we de...
[07:05:38] <wikibugs>	 10Data-Engineering, 10SRE, 10ops-eqiad: Check analytics1086 mgmt's cable - https://phabricator.wikimedia.org/T320458 (10elukey) Same thing this morning:  ` elukey@cumin1001:~$ sudo ipmitool -I lanplus -H "an-worker1086.mgmt.eqiad.wmnet" -U root -E chassis power status Unable to read password from environment...
[08:12:32] <moritzm>	 FYI; I'm switching the dse-k8s-etcd1001 VM temporarily to DRBD (to allow migration to a different node), latencies may go up temporarily 
[08:50:22] <btullis>	 moritzm: Thanks for the heads-up. 👍
[08:54:45] <icinga-wm>	 ACKNOWLEDGEMENT - DNS on an-worker1130.mgmt is CRITICAL: DNS CRITICAL - expected 0.0.0.0 but got 10.65.0.156 ayounsi https://phabricator.wikimedia.org/T320598 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[09:07:02] <moritzm>	 and switched back to "plain" after migration is completed
[09:09:03] <btullis>	 Ack, thanks. Is it just that one VM or will you be migrating the other two VMs as well?
[09:51:03] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Clement_Goubert) Hi, I'll be your SRE support for today, and will handle de/repooling, destroying th...
[10:13:20] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Clement_Goubert) Destroy/apply done in staging: ` # helmfile -e staging status helmfile.yaml: basePa...
[10:30:50] <wikibugs>	 10Analytics, 10EventStreams: Old events in the stream - https://phabricator.wikimedia.org/T320558 (10Iluvatar)
[11:16:00] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10JArguello-WMF)  @Clement_Goubert Thank you so much! Please let us know if there is anything we need...
[11:20:56] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Clement_Goubert) `eventstream` redeployed in codfw.  @JArguello-WMF Apart from checking everything i...
[11:52:21] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Clement_Goubert) `eventstream` redeployed in eqiad
[11:59:55] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Clement_Goubert) Everything looks healthy from my end, both are getting traffic and not throwing err...
[12:37:57] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams: Old events in the stream - https://phabricator.wikimedia.org/T320558 (10Ottomata) Interesting!  Adding some tags and folks.  Would like to look into this.   Indeed, if it happens after 15 or 30 minutes, something weird must be...
[12:40:44] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Ottomata) > eventstreams-internal is still used?  I am not sure!  I'd imagine folks use it, as it is...
[12:52:35] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams: Old events in the stream - https://phabricator.wikimedia.org/T320558 (10Ottomata) (BTW, thank you for the very good bug report @Iluvatar :) )
[13:10:21] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams: Old events in the stream - https://phabricator.wikimedia.org/T320558 (10Iluvatar) I tried to hardcode `EventSource(url, last_id = None)`, but without effect. I use streams since 2017, but this bug appeared only 6 days ago. Almo...
[13:32:56] <wikibugs>	 10Data-Engineering, 10Product-Analytics, 10wmfdata-python, 10Data Pipelines (Sprint 02): Upgrade WMFData Python Package to use  Spark3 - https://phabricator.wikimedia.org/T318587 (10Antoine_Quhen) linked with https://phabricator.wikimedia.org/T273210
[13:54:48] <icinga-wm>	 RECOVERY - Host an-worker1098.mgmt is UP: PING OK - Packet loss = 0%, RTA = 12.53 ms
[13:56:44] <icinga-wm>	 RECOVERY - Host an-worker1087.mgmt is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms
[13:59:26] <icinga-wm>	 RECOVERY - Host an-worker1086.mgmt is UP: PING OK - Packet loss = 0%, RTA = 0.60 ms
[13:59:47] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams: Old events in the stream - https://phabricator.wikimedia.org/T320558 (10Iluvatar) JavaScript (Browser): https://swviewer.toolforge.org/test.html (see concole):  Simple code from Wikitech example:  ` <script> const url = 'https:...
[14:01:24] <icinga-wm>	 RECOVERY - Host analytics1073.mgmt is UP: PING OK - Packet loss = 0%, RTA = 1.41 ms
[15:03:27] <icinga-wm>	 RECOVERY - DNS on an-worker1130.mgmt is OK: DNS OK: 0.016 seconds response time. an-worker1130.mgmt.eqiad.wmnet returns 10.65.0.156 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:25:07] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): Track views of EntitySchema namespaces on Wikidata (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/811979 (https://phabricator.wikimedia.org/T304793) (owner: 10Michael Große)
[15:25:44] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Remove materialized json files and disable materializing them [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/839677 (https://phabricator.wikimedia.org/T315674) (owner: 10Ottomata)
[15:26:26] <ottomata>	 !log remove materialized .json files from schemas/event/primary - this should be a no-op as no clients should actually be using the json files. - T315674
[15:26:29] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:26:29] <stashbot>	 T315674: Remove materialized .json files from event schema repositories - https://phabricator.wikimedia.org/T315674
[15:29:50] <wikibugs>	 (03Merged) 10jenkins-bot: Remove materialized json files and disable materializing them [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/839677 (https://phabricator.wikimedia.org/T315674) (owner: 10Ottomata)
[15:39:11] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Clement_Goubert) `eventstreams-internal` fully redeployed, this task can probably be closed now.
[15:42:24] <wikibugs>	 10Data-Engineering, 10SRE, 10serviceops, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Ottomata) Thank you so much!
[15:53:34] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream: Refactor EventBus extension Hooks to use new hook system - https://phabricator.wikimedia.org/T320655 (10Ottomata)
[16:06:42] <wikibugs>	 (03PS21) 10Joal: Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739)
[16:15:22] <isaacj>	 o/ I'm running what I assume is a very simple query on Hive on stat1008/stat1007 (tested both) and getting a Java memory error (`Exception in thread "main" java.lang.OutOfMemoryError: Java heap space`): `SELECT * FROM wmf.webrequest WHERE year = 2022 and month = 10 and day = 10 and hour = 18 and uri_path = "Special:Preferences" LIMIT 5;` removing the uri_path clause does not throw memory errors but I can't see anything particularly 
[16:15:22] <isaacj>	 intensive about the query. any thoughts?
[16:20:53] <elukey>	 isaacj: o/ can you try the query after `export HADOOP_HEAPSIZE=2048` ?
[16:21:24] <elukey>	 afaics it seems a problem of the hive client's jvm, that IIRC runs with 256MB of max heap space 
[16:21:40] * isaacj checking
[16:22:15] <wikibugs>	 10Analytics-Jupyter, 10Data-Engineering, 10Product-Analytics: Replace anaconda-wmf with smaller, non-stacked Conda environments - https://phabricator.wikimedia.org/T302819 (10nshahquinn-wmf) >>! In T302819#8310057, @Ottomata wrote: > Sounds fine to me!  I don't have much of an opinion, so if @aqu and @xcolla...
[16:25:38] <wikibugs>	 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Rerunning Spark functions with changed settings has no effect - https://phabricator.wikimedia.org/T273210 (10nshahquinn-wmf)
[16:27:08] <isaacj>	 elukey: yep, that seems to be working (still running). this feels like new behavior though (the need to increase heap size) -- any idea if this is a temporary condition due to high load, or if defaults were changed, or something else in the Hive backend has changed that this is required now, or if I'm just wrong in that this is actually a very expensive query? i'm also open to "don't use Hive command-line please, stick with PySpark 
[16:27:08] <isaacj>	 etc."
[16:29:38] <elukey>	 isaacj: no idea, but it feels like the extra uri_path condition causes some extra work done on the local side (so on the hive client's jvm). Joseph is surely the best person to ask for a more in depth explanation :D
[16:30:01] <elukey>	 PySpark is always the preferred choice of course :)
[16:30:18] <elukey>	 (hive runs map-reduce etc.., spark sql is more efficient)
[16:32:40] <isaacj>	 yeah, i guess i forget how big webrequest is too. back to PySpark I retreat :)
[18:24:53] <wikibugs>	 10Analytics-Jupyter, 10Data-Engineering, 10Product-Analytics: Replace anaconda-wmf with smaller, non-stacked Conda environments - https://phabricator.wikimedia.org/T302819 (10nshahquinn-wmf) Okay, another suggestion about channels! What if we set up a [custom channel](https://docs.conda.io/projects/conda/en/...
[18:24:55] <wikibugs>	 10Data-Engineering, 10SRE, 10ops-eqiad: Check analytics1086 mgmt's cable - https://phabricator.wikimedia.org/T320458 (10Jclark-ctr) a:03Jclark-ctr
[18:25:45] <wikibugs>	 10Data-Engineering, 10SRE, 10ops-eqiad: Check analytics1086 mgmt's cable - https://phabricator.wikimedia.org/T320458 (10Jclark-ctr) @BTullis  @elukey   Management switch failed today and was replaced can you verify if it is still not working for you?
[18:33:25] <wikibugs>	 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Remove Spark session timeout functionality from Wmfdata-Python - https://phabricator.wikimedia.org/T298179 (10nshahquinn-wmf)
[19:39:00] <wikibugs>	 10Data-Engineering, 10Privacy Engineering: Investigate releasing historical top-pageview-per-country data - https://phabricator.wikimedia.org/T299627 (10Htriedman) Hi all! Just wanted to come back to this thread (even though it's been more than a month or two) with some updates —   The Privacy Engineering team...
[20:05:54] <joal>	 isaacj: sorry for the delay in answering! I think the problem you've been facing with the memory on hive must have come from the job tring to optimize and run the query locally, while the un-tangling of all parquet columns is not that cheap in memory - I think that the same quwery with just a small number of fields selected would probably go without memory error
[20:14:44] <isaacj>	 oh interesting joal thanks for clarifying! that makes a lot of sense 
[20:15:19] <joal>	 isaacj: I must tell you that I'm not sure of what I'm saying, but it all seems to make sense indeed :)
[20:16:16] <isaacj>	 joal: haha well i'm adding it to my mythology of hive :)
[20:16:25] <joal>	 hehe :D