[00:05:14] RECOVERY - SSH on aqs1008.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [00:08:45] (03CR) 10Bartosz Dziewoński: [C: 03+2] EditAttemptStep: add new values for init_mechanism [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/805728 (https://phabricator.wikimedia.org/T298634) (owner: 10DLynch) [00:09:21] (03Merged) 10jenkins-bot: EditAttemptStep: add new values for init_mechanism [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/805728 (https://phabricator.wikimedia.org/T298634) (owner: 10DLynch) [00:21:20] PROBLEM - Check systemd state on an-web1001 is CRITICAL: CRITICAL - degraded: The following units failed: hardsync-published.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:30:32] RECOVERY - Check systemd state on an-web1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:25:46] PROBLEM - Check unit status of monitor_refine_event on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [01:26:14] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_event.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [05:11:49] PROBLEM - SSH on aqs1008.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [06:12:56] RECOVERY - SSH on aqs1008.mgmt is OK: SSH OK - OpenSSH_7.0 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [06:31:46] Hi btullis - Thank you for yesterday's fix on puppet - I'm sorry I didn't catch it in reviewing :( I'm that used to puppet [06:49:13] !log Rerun webrequest-load-wf-upload-2022-6-15-22 after weird oozie failure [06:49:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:47:40] Hi, whom shall I contact about gitlab-ci workers disk spaces? As I build on it, it often crashes because of insufficient disc space. [08:47:41] "E: You don't have enough free space in /var/cache/apt/archives/." [08:48:47] aqu: I had a ticket about this but it got closed - I think it should be reopened [08:49:10] All pipelines seems broken now. On a pylint job: "error: could not create 'build': No space left on device" [08:49:33] aqu: https://phabricator.wikimedia.org/T310593 [08:49:37] Thx [09:05:59] a-team [09:08:21] yes jynus? [09:08:38] there is breakage on 2 locations of D.E. infra [09:08:49] hm - ping btullis ? [09:08:55] I did, no answer [09:09:02] mwarf [09:09:04] I can help but won't do without permission [09:09:09] can ou tell me more jynus [09:09:15] please [09:09:17] +y [09:09:31] dbstore1003 is close to expliding due to a 43-day query running [09:09:41] it is research user [09:09:51] can kill the query, but wanted permission first [09:10:00] *exploding [09:10:12] 43-day-long query [09:10:20] ack - I don't think this query is ours - please kill it [09:10:24] jynus: --^ [09:10:25] ok [09:10:38] the other is an-coord, prometheus metrics went down [09:10:46] not sure if service down or maintenance or something [09:10:47] aouch [09:11:04] I can research further if not expected [09:11:13] but let me start with the kill first [09:11:19] sure [09:12:04] an-coord1001 or 1002? [09:12:08] s7 replication is flowing again [09:12:23] so that fixed it (will take some time to be back to good state) - re: dbstore1003 [09:12:34] thanks a lot for that already jynus [09:12:36] aqu: let me check that [09:13:07] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (an-coord1001:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [09:13:11] so 1001 [09:13:24] hm [09:13:26] I can check if an issue with the exporter or the server itself [09:14:09] mysql seems up [09:14:49] weird, the exporter is up too [09:15:21] and metrics seems to be flowing [09:16:03] so maybe it is a config thing- but certainly that can wait [09:17:16] dbstore1003 lag is recovering, which was the most concerning thing, so I think we are good [09:17:30] ack jynus - thanks again [09:17:42] just mention the monitoring alerm to btullis when back [09:18:01] thanks to you for the help! [09:18:49] if dbstore1003 would have kept like that for more time, it would have crashed and the recovery would have been more painfil [09:20:01] all good :) [09:20:11] this is the link to pass for further research: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&var-site=eqiad&var-group=analytics&var-shard=All&var-role=All&viewPanel=4&from=1655349592737&to=1655371192738 [09:20:31] Sorry for the delay. I'm here now. [09:20:33] either some restart or job config review needed- [09:21:11] don't worry, the most pressing issue seems to be solved [09:21:18] Thanks for the intervention jynus [09:21:32] I will leave debugging of the other for you, plenty of work left (but probably not as urgent :-)) [09:21:43] cheers! [09:22:23] (I would try first to just restart the local exporter- maybe it is stuck) [09:22:27] bye [09:30:53] The metrics seems ok for an-coord1001 and an-coord1002 at the moment, so I don't think I need to restart any exporters. [10:52:50] 10Data-Engineering, 10Data-Engineering-Kanban: Analytics Data Lake - Hadoop Namenode failure - standby namenode backups filled up namenode data partition - https://phabricator.wikimedia.org/T309649 (10BTullis) Having looked into the issue with the `hdfs dfsadmin -fetchImage` job, I'm not sure that this is the... [12:08:19] (03PS1) 10Joal: Update geoeditor HQL scripts for spark3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/806200 [12:13:01] (03CR) 10Joal: [V: 03+1] "Tested on cluster with spark3" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/806200 (owner: 10Joal) [12:24:06] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Analytics Data Lake - Hadoop Namenode failure - standby namenode backups filled up namenode data partition - https://phabricator.wikimedia.org/T309649 (10BTullis) This was discussed with @fgiunchedi in [[https://wm-bot.wmflabs.org/libera_logs... [12:34:20] 10Data-Engineering-Radar, 10Cassandra, 10Generated Data Platform, 10Patch-For-Review: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 (10Eevans) [12:37:15] 10Data-Engineering-Radar, 10Cassandra, 10Generated Data Platform, 10Patch-For-Review: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 (10Eevans) [12:53:09] 10Data-Engineering, 10Discovery: Late events in wdqs-external.sparql-query? - https://phabricator.wikimedia.org/T310790 (10Ottomata) [13:02:33] 10Data-Engineering, 10Equity-Landscape: Readership Metrics Transformation - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) ` SELECT * FROM kcv.georeadership_output_rank_metrics ` [13:03:00] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [13:03:07] 10Data-Engineering, 10Equity-Landscape: Readership Metrics Transformation - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) 05Open→03Resolved [13:44:27] 10Data-Engineering, 10Data-Engineering-Kanban: Analytics Data Lake - Hadoop Namenode failure - standby namenode backups filled up namenode data partition - https://phabricator.wikimedia.org/T309649 (10BTullis) This check is now in place and working. I will resolve this ticket. {F35246833,width=80%} [13:45:21] 10Data-Engineering, 10Data-Engineering-Kanban: Analytics Data Lake - Hadoop Namenode failure - standby namenode backups filled up namenode data partition - https://phabricator.wikimedia.org/T309649 (10BTullis) [14:06:37] 10Data-Engineering, 10Data-Engineering-Kanban: Analytics Data Lake - Hadoop Namenode failure - standby namenode backups filled up namenode data partition - https://phabricator.wikimedia.org/T309649 (10BTullis) I have marked the incident report as in-review: https://wikitech.wikimedia.org/wiki/Incidents/2022-05... [14:21:39] (03CR) 10Milimetric: Update geoeditor HQL scripts for spark3 (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/806200 (owner: 10Joal) [14:23:13] 10Data-Engineering-Kanban: Build and install spark3 assembly - https://phabricator.wikimedia.org/T310578 (10JArguello-WMF) [14:27:01] (03CR) 10Btullis: [C: 03+2] "recheck" [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/805839 (https://phabricator.wikimedia.org/T310079) (owner: 10Btullis) [14:29:58] 10Data-Engineering, 10Data-Engineering-Kanban: Analytics Data Lake - Hadoop Namenode failure - standby namenode backups filled up namenode data partition - https://phabricator.wikimedia.org/T309649 (10Ottomata) I don't know! Asking in #wikimedia-sre IRC [14:44:52] 10Data-Engineering, 10SRE, 10Traffic, 10Patch-For-Review, 10User-zeljkofilipin: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) I've run out of time to work on this for now, so I'm removing the #data-engineering-kanba... [14:52:24] (03Merged) 10jenkins-bot: Update the name of the binary used to launch datahub-frontend [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/805839 (https://phabricator.wikimedia.org/T310079) (owner: 10Btullis) [15:04:46] 10Data-Engineering-Kanban, 10Airflow: [Airflow] Migrate Oozie's mediawiki_history_load jobs to Airflow - https://phabricator.wikimedia.org/T309718 (10JArguello-WMF) [15:25:00] (03PS3) 10Btullis: Update branding for DataHub to include WMF customization [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/805408 (https://phabricator.wikimedia.org/T310629) [15:29:30] PROBLEM - AQS root url on aqs2012 is CRITICAL: connect to address 10.192.48.189 and port 7232: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS%23Monitoring [15:29:32] PROBLEM - Check systemd state on aqs2005 is CRITICAL: CRITICAL - degraded: The following units failed: aqs.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:30:04] PROBLEM - AQS root url on aqs2009 is CRITICAL: connect to address 10.192.48.186 and port 7232: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS%23Monitoring [15:30:22] PROBLEM - Check systemd state on aqs2004 is CRITICAL: CRITICAL - degraded: The following units failed: aqs.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:30:30] PROBLEM - AQS root url on aqs2006 is CRITICAL: connect to address 10.192.16.168 and port 7232: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS%23Monitoring [15:30:34] PROBLEM - Check systemd state on aqs2010 is CRITICAL: CRITICAL - degraded: The following units failed: aqs.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:30:34] PROBLEM - AQS root url on aqs2005 is CRITICAL: connect to address 10.192.16.42 and port 7232: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS%23Monitoring [15:30:36] PROBLEM - Check systemd state on aqs2012 is CRITICAL: CRITICAL - degraded: The following units failed: aqs.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:36:08] 10Data-Engineering, 10Patch-For-Review: Decide whether to migrate from Presto to Trino - https://phabricator.wikimedia.org/T266640 (10JArguello-WMF) [15:38:24] PROBLEM - AQS root url on aqs2011 is CRITICAL: connect to address 10.192.48.188 and port 7232: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS%23Monitoring [16:00:57] ottomata: Heya - thanks a lot for the refine rerun - I have a question for you on this, that's why I hadn't rerun it earlier [16:02:50] 10Analytics, 10Data-Engineering, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10odimitrijevic) [16:05:31] 10Data-Engineering-Radar, 10Cassandra, 10Generated Data Platform, 10Patch-For-Review: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 (10ayounsi) @Eevans The easiest is to look at https://librenms.wikimedia.org/bill/bill_id=24/ and each link (under Billed Ports) individua... [16:07:31] 10Data-Engineering, 10Event-Platform, 10SRE, 10serviceops: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Ottomata) a:05Jelto→03None [16:08:23] (03PS4) 10Snwachukwu: Add projectview hql scripts to analytics/refinery/hql path. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/797240 (https://phabricator.wikimedia.org/T309023) [16:08:56] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10SRE, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10JArguello-WMF) [16:09:59] 10Data-Engineering-Radar, 10Event-Platform, 10Generated Data Platform: Add Event Platform timestamp JSONSchema -> Flink type support - https://phabricator.wikimedia.org/T310495 (10JArguello-WMF) [16:12:09] 10Data-Engineering-Kanban, 10Data-Engineering-Radar, 10Event-Platform, 10Generated Data Platform, 10Patch-For-Review: Add better support for using Event Platform streams with the Flink DataStream API - https://phabricator.wikimedia.org/T310302 (10Ottomata) [16:12:47] 10Data-Engineering-Kanban, 10Data-Engineering-Radar, 10Generated Data Platform, 10Patch-For-Review: Flink output support for Event Platform events - https://phabricator.wikimedia.org/T310218 (10Ottomata) [16:14:00] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10SRE, 10serviceops: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10Ottomata) [16:14:04] 10Data-Engineering: Update webrequest error thresholds - https://phabricator.wikimedia.org/T310576 (10JAllemandou) a:03JAllemandou [16:16:45] 10Data-Engineering, 10Airflow: [Airflow] Refactor HDFSArchiveOperator to run in Skein - https://phabricator.wikimedia.org/T310542 (10JArguello-WMF) p:05Triage→03High [16:18:29] 10Data-Engineering: Airflow: pin dependency versions to prevent long installs - https://phabricator.wikimedia.org/T309046 (10Ottomata) Also, once we provide spark 3, we should make airflow-dags avoid depending on pyspark, if we can. [16:19:06] 10Data-Engineering, 10Data-Engineering-Kanban: Update webrequest error thresholds - https://phabricator.wikimedia.org/T310576 (10JAllemandou) [16:20:25] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] URLSensor might be preventing alerts to fire correctly - https://phabricator.wikimedia.org/T309563 (10JArguello-WMF) [16:22:53] 10Data-Engineering, 10GitLab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10JAllemandou) [16:23:40] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Airflow: pin dependency versions to prevent long installs - https://phabricator.wikimedia.org/T309046 (10JArguello-WMF) p:05Triage→03High [16:24:50] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Airflow: pin dependency versions to prevent long installs - https://phabricator.wikimedia.org/T309046 (10Ottomata) In the meantime, we should probably use [[ https://phabricator.wikimedia.org/phame/post/view/285/production_excellence_44_may_2022/ | ai... [16:28:40] 10Data-Engineering, 10MediaWiki-General: Pingback dashboard data normalisation - https://phabricator.wikimedia.org/T298928 (10JArguello-WMF) a:03Milimetric [16:29:08] 10Data-Engineering, 10MediaWiki-General: Pingback dashboard data normalisation - https://phabricator.wikimedia.org/T298928 (10JArguello-WMF) 05Open→03Resolved [16:34:47] 10Data-Engineering-Radar, 10Platform Engineering: Deploy AQS service to codfw clusters - https://phabricator.wikimedia.org/T309808 (10JArguello-WMF) [16:35:16] (03PS2) 10Joal: Update geoeditor HQL scripts for spark3 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/806200 [16:36:09] (03CR) 10Joal: Update geoeditor HQL scripts for spark3 (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/806200 (owner: 10Joal) [16:44:04] 10Data-Engineering, 10Gerrit: Remove unused Gerrit repository mediawiki/services/aqs/deploy - https://phabricator.wikimedia.org/T309731 (10Milimetric) 05Open→03Resolved a:03Milimetric Sorry, Andre, I didn't even know there was a Gerrit tag. I'm marking this as resolved for now. If we ever come up with... [16:46:27] 10Data-Engineering: Event Utilities partially downloads schemas - https://phabricator.wikimedia.org/T309717 (10JArguello-WMF) p:05Triage→03Low a:03Ottomata [16:50:37] 10Data-Engineering, 10Cassandra: Encrypt Spark-Cassandra connection - https://phabricator.wikimedia.org/T310820 (10JAllemandou) [16:51:21] 10Data-Engineering, 10Cassandra: Make Cassandra client encryption non-optional (AQS cluster) - https://phabricator.wikimedia.org/T309229 (10JAllemandou) Question about AQS Should we wait for AQS-2.0 to do this instead of changing the old node code? [16:51:43] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for kcgwiki - https://phabricator.wikimedia.org/T305280 (10JArguello-WMF) a:03BTullis [16:53:11] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for kcgwiki - https://phabricator.wikimedia.org/T305280 (10JArguello-WMF) [16:53:29] 10Data-Engineering-Radar, 10Cassandra: Make Cassandra client encryption non-optional (AQS cluster) - https://phabricator.wikimedia.org/T309229 (10JAllemandou) [16:54:52] 10Data-Engineering-Radar, 10Cassandra: Make Cassandra client encryption non-optional (AQS cluster) - https://phabricator.wikimedia.org/T309229 (10Eevans) >>! In T309229#8009588, @JAllemandou wrote: > Question about AQS Should we wait for AQS-2.0 to do this instead of changing the old node code? We could, that... [16:55:17] 10Data-Engineering-Radar, 10Cassandra: Make Cassandra client encryption non-optional (AQS cluster) - https://phabricator.wikimedia.org/T309229 (10Ottomata) cc @tchin who is investigating Flink -> Cassandra in {T306627} [16:58:14] (03PS1) 10Milimetric: Add kcgwiki to the sqoop list [analytics/refinery] - 10https://gerrit.wikimedia.org/r/806244 (https://phabricator.wikimedia.org/T305280) [16:59:16] (03CR) 10Milimetric: "Ben, if you do the prepare views task (linked here), just ping me and I'll merge this." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/806244 (https://phabricator.wikimedia.org/T305280) (owner: 10Milimetric) [17:00:56] 10Data-Engineering-Radar, 10Cassandra, 10Generated Data Platform, 10Patch-For-Review: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 (10Eevans) >>! In T307641#8009415, @ayounsi wrote: > @Eevans > The easiest is to look at https://librenms.wikimedia.org/bill/bill_id=24/ a... [17:03:22] 10Data-Engineering-Radar, 10Cassandra, 10Generated Data Platform, 10Patch-For-Review: AQS multi-datacenter cluster expansion - https://phabricator.wikimedia.org/T307641 (10Eevans) [17:08:46] 10Data-Engineering-Icebox, 10Data-Engineering-Kanban, 10Event-Platform, 10Metrics-Platform, and 2 others: Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10JArguello-WMF) [17:09:55] 10Data-Engineering, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge, 10Performance-Team (Radar): Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10JArguello-WMF) [17:28:16] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Update HiveToCassandra job to read cassandra password from file - https://phabricator.wikimedia.org/T306895 (10JArguello-WMF) a:05NOkafor-WMF→03None [17:29:42] 10Data-Engineering, 10Data-Services, 10Patch-For-Review: Move wikireplicas dbproxy haproxy config to etcd - https://phabricator.wikimedia.org/T304478 (10JArguello-WMF) a:05BTullis→03None [17:34:50] 10Data-Engineering: Automatically monitor schema changes that would break sqoop - https://phabricator.wikimedia.org/T310824 (10Milimetric) [17:35:58] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Airflow DagProcessor not refreshing all dags - https://phabricator.wikimedia.org/T310297 (10JArguello-WMF) 05Open→03Resolved [17:38:08] 10Data-Engineering, 10Event-Platform, 10Metrics-Platform, 10Browser-Support-Microsoft-Edge, 10Performance-Team (Radar): Problem with delay caused by intake-analytics.wikimedia.org - https://phabricator.wikimedia.org/T295427 (10JArguello-WMF) [17:38:34] 10Data-Engineering-Icebox, 10DBA, 10Data-Services, 10Patch-For-Review, 10cloud-services-team (Kanban): Prepare and check storage layer for kcgwiki - https://phabricator.wikimedia.org/T305280 (10JArguello-WMF) [17:53:09] (03PS1) 10Jiyu: Prettify User not found page [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/806271 (https://phabricator.wikimedia.org/T134661) [18:25:42] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10SRE, 10serviceops: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10SLyngshede-WMF) p:05Triage→03Medium [18:37:26] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Define the Helm charts and helmfile deployments for Datahub - https://phabricator.wikimedia.org/T301454 (10JMeybohm) [18:38:10] 10Data-Engineering, 10Data-Catalog, 10SRE, 10serviceops, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) 05Open→03Resolved All merged. Thanks! 🎉 [20:44:53] (03CR) 10Vivian Rook: [C: 03+1] "This looks good. Though the error page does not seem to recognize that I am logged in. It offers me a login button which leads me back to " [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/806271 (https://phabricator.wikimedia.org/T134661) (owner: 10Jiyu) [20:47:27] 10Data-Engineering-Kanban, 10Data-Engineering-Radar, 10Event-Platform, 10Generated Data Platform: Add better support for using Event Platform streams with the Flink DataStream API - https://phabricator.wikimedia.org/T310302 (10Ottomata) Alright, @gmodena @dcausse https://gerrit.wikimedia.org/r/804614 is re... [21:04:08] (03CR) 10Jiyu: Prettify User not found page (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/806271 (https://phabricator.wikimedia.org/T134661) (owner: 10Jiyu) [21:06:20] (03CR) 10Vivian Rook: [C: 03+2] Prettify User not found page [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/806271 (https://phabricator.wikimedia.org/T134661) (owner: 10Jiyu) [21:10:09] (03Merged) 10jenkins-bot: Prettify User not found page [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/806271 (https://phabricator.wikimedia.org/T134661) (owner: 10Jiyu) [23:05:42] 10Data-Engineering-Icebox: Improve Bot Detection Heuristics - https://phabricator.wikimedia.org/T310846 (10odimitrijevic) [23:18:51] 10Data-Engineering-Icebox, 10SRE, 10Traffic-Icebox: We are not capturing IPs of original requests for proxied requests from operamini and googleweblight. x-forwarded-for is null and client-ip is the same as IP on Webrequest data - https://phabricator.wikimedia.org/T232795 (10odimitrijevic)