[00:08:43] <wikibugs>	 10Analytics, 10Anti-Harassment, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for AHT - https://phabricator.wikimedia.org/T226853 (10Niharika) >>! In T226853#5301929, @nettrom_WMF wrote: > @Niharika : Does AHT have any EventLogging schemas that are whit...
[01:28:46] <wikibugs>	 (03PS1) 10Milimetric: Improve example run command [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520661
[01:41:12] <wikibugs>	 10Analytics, 10good first bug: [reportupdater] Allow defaults for all config parameters - https://phabricator.wikimedia.org/T193171 (10Milimetric) I got a ping from @Geekbug, will update description here to link to ReportUpdater docs and source.  Feel free to ping on IRC again.
[01:41:47] <wikibugs>	 10Analytics, 10good first bug: [reportupdater] Allow defaults for all config parameters - https://phabricator.wikimedia.org/T193171 (10Milimetric)
[01:42:33] <milimetric>	 gonna peek in a little later at the checker job, but if it runs fine and Europe folks are working tomorrow, yall can point AQS at the new snapshot and deploy it.
[02:31:58] <milimetric>	 well, looks like Druid indexing failed
[02:32:56] <milimetric>	 I feel like I might make a mistake if I try to figure out why right now, so I'm gonna wait until tomorrow
[03:09:37] <wikibugs>	 10Analytics, 10Anti-Harassment, 10Product-Analytics: Hash all pageTokens or temporary identifiers from the EL Sanitization white-list for AHT - https://phabricator.wikimedia.org/T226853 (10dmaza)  >>! In T226853#5305428, @Niharika wrote: >>>! In T226853#5301929, @nettrom_WMF wrote: >> @Niharika : Does AHT ha...
[04:01:19] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Chinese-Sites: X-axis is at odds with stated period in header of trend charts for 'total articles' for a wiki - https://phabricator.wikimedia.org/T180118 (10Shizhao)
[04:30:46] <wikibugs>	 (03CR) 10Nuria: [C: 03+2] Improve example run command [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520661 (owner: 10Milimetric)
[04:37:24] <wikibugs>	 (03PS1) 10Nuria: Special:ConfirmEmail should not be a pageview [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/520671 (https://phabricator.wikimedia.org/T226730)
[06:39:07] <elukey>	 good morning :)
[06:39:17] * elukey looks for joal but doesn't find him sigh
[06:39:26] <elukey>	 :D
[07:01:51] <joal>	 Good morning elukey :)
[07:01:59] <joal>	 You won't find me every morning though )
[07:05:22] <elukey>	 bonjourrrr
[07:05:24] <elukey>	 :)
[07:10:25] <elukey>	 joal: I am playing with spark settings in the test cluster to specify a range of ports  for the driver, to enable base firewall on the stat/notebook hosts
[07:10:38] <joal>	 wow
[07:10:41] <joal>	 tricky !
[07:12:32] <elukey>	 joal: the trick is in https://spark.apache.org/docs/2.3.1/configuration.html#networking
[07:12:44] <elukey>	 (easy setting to make ops' life easy)
[07:12:52] <elukey>	 "Maximum number of retries when binding to a port before giving up. When a port is given a specific value (non 0), each subsequent retry will increment the port used in the previous attempt by 1 before retrying. This essentially allows it to try a range of ports from the start port specified to port + maxRetries."
[07:14:22] <joal>	 WOah! Nice
[07:14:40] <joal>	 Controlled round-robin ports :)
[07:16:23] <joal>	 ok, I found the issue with mediawiki_history_reduced - In the patch I made to use pageFirstEditTimestamp page page-create events I forgot a case in my test
[07:18:20] <wikibugs>	 (03CR) 10Joal: "Commenting about the error seen in June snapshot job." (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/519349 (https://phabricator.wikimedia.org/T221825) (owner: 10Joal)
[07:20:20] <wikibugs>	 (03PS1) 10Joal: Correct mediawiki-history-reduced generation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520692
[07:21:43] <joal>	 elukey: Asking for permission to update mediawiki-history-reduced prod job with patch above, and rerun it
[07:22:27] <elukey>	 joal: do you mean manually or deploying?
[07:22:34] <elukey>	 (in any case yes :)
[07:22:40] <elukey>	 (I am just curious)
[07:24:18] <joal>	 manuallky :)
[07:29:13] <elukey>	 the spark port range seems to work
[07:29:17] <elukey>	 on an-tool1006
[07:29:22] <elukey>	 (with ferm enabled)
[07:30:51] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Patch-For-Review, 10User-Elukey: Enable base::firewall on stat boxes after restricting Spark REPL ports. - https://phabricator.wikimedia.org/T170826 (10elukey) an-tool1006 seems to work fine with the new settings and ferm enabled! I opened pyspark2 and spark2-shell (both...
[07:56:54] <joal>	 elukey: sorry I had to leave, people at the door
[07:56:59] <joal>	 elukey: ok for manual patch?
[07:57:15] <elukey>	 sure!
[07:57:23] <joal>	 thanks mate :)
[08:00:30] <joal>	 !log Kill mediawiki-history-redeuced coordinator and restart it with manually patched version
[08:00:31] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[08:12:05] <wikibugs>	 10Analytics, 10User-Elukey: Move  refinery to hive 2 actions - https://phabricator.wikimedia.org/T227257 (10elukey)
[08:15:27] <elukey>	 this will be long --^
[09:05:09] <wikibugs>	 10Analytics, 10Analytics-Cluster, 10Patch-For-Review, 10User-Elukey: Enable base::firewall on stat boxes after restricting Spark REPL ports. - https://phabricator.wikimedia.org/T170826 (10elukey) I checked via netstat all the ports opened on stat boxes, and I have a few comments:  1) People seems to use a...
[09:06:55] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Allow all Analytics tools to work with Kerberos auth - https://phabricator.wikimedia.org/T226698 (10elukey)
[10:15:14] <wikibugs>	 10Analytics, 10Patch-For-Review: Review Bacula home backups set for stat100[56] - https://phabricator.wikimedia.org/T201165 (10elukey) Just sent an email to people working on stat/notebook nodes to make sure that we have a shared understanding about home directories not backed up. Also added a note in https://...
[10:27:41] <wikibugs>	 10Analytics, 10Analytics-SWAP, 10Product-Analytics: Enable widgets on Jupyter Labs on SWAP - https://phabricator.wikimedia.org/T227217 (10Neil_P._Quinn_WMF)
[10:58:20] * elukey lunch!
[13:08:58] <jbond42>	 elukey: hi luca can you check https://gerrit.wikimedia.org/r/c/operations/puppet/+/520747 specificly the kafaka ones to make sure i have picked the best url
[13:10:02] <elukey>	 jbond42: looks good, even the vk ones
[13:10:03] <elukey>	 thanks!
[13:10:21] <jbond42>	 awesome thank
[13:10:25] <elukey>	 (stepping afk for a bit)
[13:23:53] <icinga-wm>	 RECOVERY - Check if the Hadoop HDFS Fuse mountpoint is readable on an-tool1006 is OK: CRITICAL
[13:34:05] <elukey>	 nice!
[14:38:29] <wikibugs>	 10Analytics, 10Operations, 10Traffic: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10ema) 05Open→03Resolved a:03ema The maximum allowed request header size (field name + value) is now 8192 bytes. Closing.
[14:38:31] <wikibugs>	 10Analytics-Kanban, 10Patch-For-Review: Fix failing webrequest hours (upload and text 2018-06-14-11) - https://phabricator.wikimedia.org/T197281 (10ema)
[14:51:07] <wikibugs>	 10Analytics, 10Cloud-Services, 10observability, 10Patch-For-Review, and 2 others: High Prometheus TCP retransmits - https://phabricator.wikimedia.org/T225296 (10elukey) I filed a patch to add the missing PTR/AAAA records for an-coord and an-worker* hosts. After it is reviewed/merged, I'll start to roll out...
[15:14:07] <wikibugs>	 10Analytics, 10Reading Depth, 10Readers-Web-Backlog (Tracking): [Bug] Many ReadingDepth validation errors logged - https://phabricator.wikimedia.org/T216063 (10phuedx) In the last 24 hour period, 11 more ReadingDepth events with odd URLs have caused processing errors. See https://logstash.wikimedia.org/goto/...
[15:36:22] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 2 misc nodes for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10elukey)
[15:37:18] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 2 misc nodes for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10elukey)
[15:47:50] <elukey>	 fdans: o/
[15:48:02] <fdans>	 helloooo
[15:48:06] <elukey>	 I guess that today we are only two people working?
[15:48:43] <fdans>	 oh no they left me alone with elukey
[15:49:11] <elukey>	 :D
[15:49:14] <fdans>	 elukey: let's say hi to each other at 6pm?
[15:49:17] <fdans>	 :)
[15:49:34] <elukey>	 sure! Then we can surely skip grooming
[16:04:08] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] Correct mediawiki-history-reduced generation [analytics/refinery] - 10https://gerrit.wikimedia.org/r/520692 (owner: 10Joal)
[16:04:33] <milimetric>	 joal: thanks for the reduced fix, I figured it was nulls but still have no idea how to debug druid indexing jobs
[16:05:07] <milimetric>	 I'll monitor the patched job, but I've merged the fix too so I can deploy if needed
[16:05:32] <milimetric>	 also verified the old coordinator was killed, thanks!
[16:37:34] <wikibugs>	 10Analytics, 10Analytics-Kanban: Decide: start_timestamp for mediawiki history - https://phabricator.wikimedia.org/T220507 (10Neil_P._Quinn_WMF) >>! In T220507#5298048, @Milimetric wrote: > Quick note that we tried to do what we proposed here but it complicated other parts of the data too much.  So we reverted...
[16:39:57] * elukey off!
[17:01:12] <wikibugs>	 (03PS2) 10Nuria: Most special pages should not be pageviews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/520671 (https://phabricator.wikimedia.org/T226730)
[17:01:57] <wikibugs>	 (03PS3) 10Nuria: Most special pages should not be pageviews [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/520671 (https://phabricator.wikimedia.org/T226730)
[18:01:44] <wikibugs>	 10Analytics, 10Operations, 10hardware-requests, 10User-Elukey: eqiad: 2 misc nodes for the Kerberos KDC service - https://phabricator.wikimedia.org/T227288 (10MoritzMuehlenhoff) Should these really be both in eqiad? The initial use case is for analytics, but we might very well come up with a use case outsi...
[19:37:43] <wikibugs>	 (03CR) 10Awight: [C: 04-1] "It's much nicer now that the random checks live together!  Just one potential blocker, I think there's an edge case that will sneak throug" (036 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/520671 (https://phabricator.wikimedia.org/T226730) (owner: 10Nuria)
[19:44:53] <wikibugs>	 10Analytics, 10good first bug: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604 (10Ae_kaash) Hi @mforns , I am a first time user of phab. I was looking at this task. In my opinion - we can change the code such that report updater writes the c...