[00:05:56] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:15:15] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:20:06] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 51485 bytes in 2.161 second response time
[00:25:24] <wikibugs>	 10Project-Admins, 10PM: Mass-migrate project tags for parsing team - https://phabricator.wikimedia.org/T245868 (10ssastry) >>! In T245868#5929537, @Aklapper wrote: >>>! In T245868#5913009, @ssastry wrote: >> I don't think the #Services applies any more for Parsoid since it is part of core. > @ssastry: Would yo...
[00:25:53] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 91332 bytes in 7.358 second response time
[00:41:57] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:42:37] <shinken-wm>	 PROBLEM - Host deployment-dumps-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (172.16.4.101)
[01:32:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-mediawiki-09 PHP7 has broken cache of codebase? - https://phabricator.wikimedia.org/T219242 (10Krinkle) 05Open→03Declined
[01:34:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Deployment services), 10Scap, 10Operations, 10serviceops: On beta, scap can't clear opcache on some mw servers - https://phabricator.wikimedia.org/T237033 (10Krinkle) Confirmed this is still happening on every beta deploy ([latest](https://integr...
[01:34:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Deployment services), 10Scap, 10Operations, 10serviceops: Sap can't clear opcache on mw servers in Beta Cluster - https://phabricator.wikimedia.org/T237033 (10Krinkle)
[01:35:05] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Deployment services), 10Scap, 10Operations, 10serviceops: Scap can't clear opcache on mw servers in Beta Cluster - https://phabricator.wikimedia.org/T237033 (10Krinkle)
[01:35:10] <wikibugs>	 10Scap: Purge the hhvm fcgi and cli bytecache as part of deployment - https://phabricator.wikimedia.org/T146226 (10Krinkle) 05Open→03Declined
[01:35:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Scap: Argument 2 passed to WatchedItemStore::__construct() must be an instance of JobQueueGroup, instance of HashBagOStuff given, called in /srv/mediawiki/php-master/includes/ServiceWiring.php on line 610 - https://phabricator.wikimedia.org/T217942 (10Krinkle) 05Open→03Resol...
[02:48:48] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Parsoid, 10Core Platform Team Workboards (Clinic Duty Team), 10User-Ryasmeen: Parsoid-PHP should be publicly accessible in beta - https://phabricator.wikimedia.org/T247589 (10cscott) 05Open→03Resolved Restbase fixed with https://github.com/wikimedia/restbase/pull/1248
[02:48:51] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Parsoid: Replace deployment-mediawiki-parsoid10 with a "purer" deployment-parsoid11 box - https://phabricator.wikimedia.org/T246854 (10cscott)
[02:48:54] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Core Platform Team, 10Parsoid, 10Patch-For-Review, 10User-Ryasmeen: Parsoid/RESTbase seems to be unavailable in Beta - https://phabricator.wikimedia.org/T246833 (10cscott)
[03:36:36] <wikibugs>	 (03PS1) 10Jforrester: Switch example image from old node6 image to current node10 one [blubber] - 10https://gerrit.wikimedia.org/r/580167
[03:38:12] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Switch example image from old node6 image to current node10 one [blubber] - 10https://gerrit.wikimedia.org/r/580167 (owner: 10Jforrester)
[03:48:01] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-General-or-Unknown, 10PHP 7.4 support: Make Wikimedia Production MediaWiki compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247658 (10Jdforrester-WMF)
[03:51:22] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Lexicographical data, 10Wikidata, 10PHP 7.4 support: Make WikibaseLexeme compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247806 (10Jdforrester-WMF)
[03:52:02] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-General-or-Unknown, 10PHP 7.4 support: Make Wikimedia Production MediaWiki compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247658 (10Jdforrester-WMF)
[09:05:13] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Lexicographical data, 10Wikidata, 10PHP 7.4 support: Make WikibaseLexeme compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247806 (10Daimona)
[09:05:47] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Lexicographical data, 10Wikidata, 10PHP 7.4 support: Make WikibaseLexeme compatible with PHP 7.4 - https://phabricator.wikimedia.org/T247806 (10Daimona) Not because of PHP 7.4 :-[
[09:08:40] <wikibugs>	 10MediaWiki-Codesniffer: Create a sniff to enforce alphabetical order of literal arrays - https://phabricator.wikimedia.org/T247813 (10Nikerabbit)
[10:09:07] <wikibugs>	 10MediaWiki-Codesniffer: Avoid assignment in return statements - https://phabricator.wikimedia.org/T170332 (10Nikerabbit) This results in extra boilerplate and confusion whether to use local or member variable in code such as this: ` if ( $this->cache ) {   return $this->cache; }  // Long code that defines $cach...
[10:18:59] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Analytics, 10Analytics-Kanban, 10Release Pipeline, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10akosiaris) >>! In T238658#5973616, @colewhite wrote: >>>! In T238658#5972237, @akosiaris wrote: >> Sure it...
[11:43:20] <wikibugs>	 10MediaWiki-Codesniffer: Avoid assignment in return statements - https://phabricator.wikimedia.org/T170332 (10thiemowmde) How I would do it: `lang=php if ( !$this->cache ) {     $this->cache = $this->longCodeThatDefinesCache(); }  return $this->cache; `  Your specific example: `lang=php private function getGithu...
[12:25:13] <wikibugs>	 10Gerrit, 10DBA: Investigate Gerrit troubles to reach the MariaDB database - https://phabricator.wikimedia.org/T247591 (10hashar) I have checked logstash again, the issue has not occurred since March 11.
[12:27:18] <wikibugs>	 10Gerrit, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when cloning repos - https://phabricator.wikimedia.org/T246763 (10hashar) I have another task about...
[14:27:21] <hashar>	  ! [remote rejected]       HEAD -> refs/for/wmf/1.35.0-wmf.24 (implicit merges detected)
[14:27:22] <hashar>	 yeahhh
[15:14:01] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T233872 (10hashar) The patch for the GlobalBlocking extension has been merged/released ( T229731 ). I have dropped it from our...
[15:20:43] <wikibugs>	 10MediaWiki-Codesniffer, 10User-DannyS712: Update UnrecognizedAnnotation following new stable interface policy - https://phabricator.wikimedia.org/T247836 (10DannyS712)
[15:21:00] <wikibugs>	 10MediaWiki-Codesniffer, 10User-DannyS712: Update UnrecognizedAnnotation following new stable interface policy - https://phabricator.wikimedia.org/T247836 (10DannyS712) See {T193613} for history
[15:23:25] <wikibugs>	 (03PS1) 10DannyS712: Update UnrecognizedAnnotation sniff with new tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/580356 (https://phabricator.wikimedia.org/T247836)
[15:24:58] <wikibugs>	 (03PS2) 10DannyS712: Update UnrecognizedAnnotation sniff with new tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/580356 (https://phabricator.wikimedia.org/T247836)
[15:26:31] <wikibugs>	 (03PS3) 10DannyS712: Update UnrecognizedAnnotation sniff with new tags [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/580356 (https://phabricator.wikimedia.org/T247836)
[15:29:26] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T233872 (10DannyS712) >>! In T233872#5975662, @hashar wrote: > The patch for the GlobalBlocking extension has been merged/rele...
[15:39:32] <hashar>	 brennen: ahhh
[15:39:39] <hashar>	 I forgot to reset --hard my commit on deployment
[15:39:53] <brennen>	 no worries.
[15:39:56] <hashar>	 sorry bout that and thanks to the comment above
[15:40:26] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T233872 (10hashar) @DannyS712 indeed, thanks to have noticed!
[15:40:40] <hashar>	 so you can do the promote again
[15:40:46] <hashar>	 or we can just bump enwiki and see what happens
[16:12:23] <wikibugs>	 (03PS1) 10Jforrester: layout: Run PHP74 for nonselenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/580366
[16:14:16] <wikibugs>	 (03PS1) 10Jforrester: layout: [mediawiki/extensions/CirrusSearch] Run PHP74 too [integration/config] - 10https://gerrit.wikimedia.org/r/580367
[16:14:18] <wikibugs>	 (03PS1) 10Jforrester: layout: Drop extension-quibble-not-php74, unused [integration/config] - 10https://gerrit.wikimedia.org/r/580368
[16:15:36] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: Run PHP74 for nonselenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/580366 (owner: 10Jforrester)
[16:16:32] <wikibugs>	 (03Merged) 10jenkins-bot: layout: Run PHP74 for nonselenium jobs [integration/config] - 10https://gerrit.wikimedia.org/r/580366 (owner: 10Jforrester)
[16:18:11] <James_F>	 !log Zuul: Run PHP74 for nonselenium jobs
[16:18:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:29:26] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki/extensions/CirrusSearch] Run PHP74 too [integration/config] - 10https://gerrit.wikimedia.org/r/580367 (owner: 10Jforrester)
[16:30:46] <wikibugs>	 (03Merged) 10jenkins-bot: layout: [mediawiki/extensions/CirrusSearch] Run PHP74 too [integration/config] - 10https://gerrit.wikimedia.org/r/580367 (owner: 10Jforrester)
[16:31:02] <James_F>	 !log Zuul: Run PHP74 for CirrusSearch again
[16:31:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:02:07] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] layout: Drop extension-quibble-not-php74, unused [integration/config] - 10https://gerrit.wikimedia.org/r/580368 (owner: 10Jforrester)
[17:05:21] <wikibugs>	 (03Merged) 10jenkins-bot: layout: Drop extension-quibble-not-php74, unused [integration/config] - 10https://gerrit.wikimedia.org/r/580368 (owner: 10Jforrester)
[17:24:18] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10ChangeProp, 10Operations, and 6 others: Migrate cpjobqueue to kubernetes - https://phabricator.wikimedia.org/T220399 (10eprodromou)
[17:43:27] <wikibugs>	 10Release-Engineering-Team, 10Repository-Admins, 10Product-Analytics (Kanban): Create a repository and user for Product Analytics Oozie jobs? - https://phabricator.wikimedia.org/T230743 (10kzimmerman) a:05kzimmerman→03mpopov
[17:55:28] <wikibugs>	 10Beta-Cluster-Infrastructure: https://commons.wikimedia.beta.wmflabs.org redirects to https://www.wikidata.org/wiki/Main_Page - https://phabricator.wikimedia.org/T247882 (10Mholloway)
[17:56:27] <wikibugs>	 10Beta-Cluster-Infrastructure: https://commons.wikimedia.beta.wmflabs.org redirects to https://www.wikidata.org/wiki/Main_Page - https://phabricator.wikimedia.org/T247882 (10Mholloway)
[17:56:58] <wikibugs>	 10Beta-Cluster-Infrastructure: https://commons.wikimedia.beta.wmflabs.org redirects to https://www.wikidata.org/wiki/Main_Page - https://phabricator.wikimedia.org/T247882 (10Reedy)
[17:57:00] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Cache, 10User-zeljkofilipin: Main pages of several Beta Cluster wikis redirect to other production wikis - https://phabricator.wikimedia.org/T247078 (10Reedy)
[18:05:13] <wikibugs>	 (03PS4) 10Hashar: Compress MediaWiki Junit XML files [integration/config] - 10https://gerrit.wikimedia.org/r/578880
[18:06:02] <wikibugs>	 (03CR) 10Hashar: "Fixed a typo in the commit message and added the list of affected jobs. I will deploy a few later today and check they are working as inte" [integration/config] - 10https://gerrit.wikimedia.org/r/578880 (owner: 10Hashar)
[18:16:26] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10MBinder_WMF)
[18:24:56] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Reedy) https://phabricator.wikimedia.org/feed/?userPHIDs=PHID-USER-mz7apcdfwd246rn55muf (plus the next page - https://phabricator.wikimedia.org/feed/?userPHIDs=PHID-USER-mz7apcdfwd246rn55muf&after=6804829274907359678...
[18:27:50] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Reedy) There's very few tasks left in #wmde-tech-communication-source-code-berlin -  https://phabricator.wikimedia.org/maniphest/query/AeVnaXqpQKvO/#R  It looks like #wmde-tech-communication-source-code-berlin has be...
[18:31:33] <Reedy>	 >Query (of class "ConpherenceTransactionQuery") overheated: examined more than 1,010 raw rows without finding 101 visible objects.
[18:31:35] <Reedy>	 gj Phab
[18:33:11] <hauskatze>	 Reedy: didn't we disable conph already?
[18:33:12] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Reedy) Oh, hang on. Herald re-added it  {F31687633 size=full}  It would look like it's all cleaned up TBH
[18:33:18] <Reedy>	 I dunno
[18:33:24] <Reedy>	 But I shouldn't be able to click a link and see that
[18:34:21] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10MBinder_WMF) Using the user's history page text, and a spreadsheet, I pulled out some IDs to investigate:  T220459, T220955, T221233, T221681, T222159, T222319, T222322, T224139, T224142, T224147, T224214, T225172, T...
[18:35:18] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10MBinder_WMF) @Ramsey-WMF , can you confirm @Reedy 's assessment? Anything stand out to you?
[18:36:28] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Reedy) One thing to check is what herald rule is adding that. For example T231917 did get the extra project removed, but herald didn't re-add the one removed
[18:37:07] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] Compress MediaWiki Junit XML files [integration/config] - 10https://gerrit.wikimedia.org/r/578880 (owner: 10Hashar)
[18:37:52] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Reedy) >>! In T247891#5977072, @Reedy wrote: > One thing to check is what herald rule is adding that. For example T231917 did get the extra project removed, but herald didn't re-add the one removed  Which is https://...
[18:43:58] <mayakpwiki>	 help! hi team, can you please provide access to Rummana Yasmeen to be able to ssh to "deployment-eventlog05.eqiad.wmflabs" 
[18:43:58] <mayakpwiki>	 she's a QA on the technology team and needs to be able to help product-analytics team with QAing the data in all-event.log file in beta cluster
[18:44:33] <Reedy>	 mayakpwiki: That hostname isn't right
[18:44:51] <Reedy>	 uh, apparently it is
[18:44:51] <Reedy>	 o_0
[18:45:34] <mayakpwiki>	 wikimedia-cloud asked to me reach out here for the access. pls let me know if you need any other information
[18:45:35] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10MBinder_WMF) @Reedy I believe that is because the Herald is set to only add the project once. That is to stop it from constantly re-adding tasks that were intended to have the tag removed. That, however, doesn't acco...
[18:46:55] <Reedy>	 mayakpwiki: What username?
[18:48:10] <mayakpwiki>	 can you try ryasmeen  ? 
[18:48:26] <mayakpwiki>	 she isnt on IRC, which is why I am messaging on behalf of her. sorry about that
[18:49:23] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Ramsey-WMF) Yeah Herald missed re-adding tags for a good number of the tickets but JJMC89 appears to be going through and fixing that manually right now 😺
[18:50:17] <wikibugs>	 10Phabricator: Please create a tag for PHP 8.0 support - https://phabricator.wikimedia.org/T247895 (10MaxSem)
[18:53:07] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Analytics, 10Analytics-Kanban, 10Release Pipeline, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) @akosiaris when you find have a moment, I'm trying to set `debug.enabled=true` on the eventstrea...
[18:53:43] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Bugreporter) See https://wikitech.wikimedia.org/wiki/Phabricator#Revert_all_activity_of_a_given_user for a script to mass-rollback, but can only be run by system administrator (@Aklapper for notice).
[18:53:55] <wikibugs>	 10Phabricator: Please create a tag for PHP 8.0 support - https://phabricator.wikimedia.org/T247895 (10Reedy) 05Open→03Resolved a:03Reedy https://phabricator.wikimedia.org/project/view/4652/
[18:55:22] <Reedy>	 mayakpwiki: I've added her... Takes a while for puppet to run
[18:55:31] <mayakpwiki>	 thanks so much !!
[18:59:54] <thcipriani>	 !log re-enabling puppet agent on deployment-mediawiki-07 The last Puppet run was at Mon Feb 24 18:26:55 UTC 2020 (31711 minutes ago). Puppet is disabled. reason not specified -- nobody on the machine
[18:59:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:00:10] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10JJMC89) 05Open→03Resolved a:03JJMC89 I've restored #structured-data-backlog where Herald didn't.
[19:00:53] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Reedy) >>! In T247891#5977169, @Bugreporter wrote: > See https://wikitech.wikimedia.org/wiki/Phabricator#Revert_all_activity_of_a_given_user for a script to mass-rollback, but can only be run by system administrator...
[19:02:16] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10MBinder_WMF) Thanks @JJMC89 !
[19:03:54] <wikibugs>	 10Phabricator: Please create a tag for PHP 8.0 support - https://phabricator.wikimedia.org/T247895 (10DannyS712) Is 7.5 going to be needed?
[19:05:37] <wikibugs>	 10Phabricator: Please create a tag for PHP 8.0 support - https://phabricator.wikimedia.org/T247895 (10Reedy) >>! In T247895#5977212, @DannyS712 wrote: > Is 7.5 going to be needed?  Googling doesn't seem to find any mentions of 7.5, only 8.0 due September 2021...
[19:08:34] <wikibugs>	 10Phabricator: Please create a tag for PHP 8.0 support - https://phabricator.wikimedia.org/T247895 (10Reedy) And 7.4 is supported for a year or so after 8.0 is released - https://www.php.net/supported-versions.php
[19:15:13] <wikibugs>	 10Phabricator: Please create a tag for PHP 8.0 support - https://phabricator.wikimedia.org/T247895 (10DannyS712) Okay, I've removed 7.5 from https://phabricator.wikimedia.org/project/manage/4107/
[19:15:38] <bawolff>	 wtf, why does beta wiki have the main page set to wikidata?
[19:17:15] <bawolff>	 I guess this is https://phabricator.wikimedia.org/T247078
[19:18:17] <mutante>	 rotation to ensure sister projects also get exposure?  scnr. yea that ticket looks like it
[19:19:31] <James_F>	 bawolff: Things are broken, yes.
[19:19:44] <James_F>	 bawolff: If you fix it, you own it. ;-)
[19:19:49] <bawolff>	 mutante: Wikinews' time will come again!
[19:19:58] <bawolff>	 James_F: oh shit, well I'll immediately stop debugging then :P
[19:20:02] <James_F>	 Next time the year starts with a '1'.
[19:20:05] * James_F laughs.
[19:20:32] <mutante>	 bawolff: ;) hehe
[19:20:41] <mutante>	 from the ticket "The whole Beta cluster? Well, not entirely… one small indomitable page still holds out—"
[19:21:31] <bawolff>	 Its definitely message corruption, the sidebar link also leads to wikidata
[19:27:12] <bawolff>	 although, i don't even know how that would result in a wikidata interwiki
[19:28:47] <bawolff>	 https://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&meta=siteinfo
[19:28:52] <bawolff>	 well i guess that's how
[19:32:48] <Reedy>	 rofl
[19:34:33] <James_F>	 Config breakage?
[19:34:38] <James_F>	 Or code breakage?
[19:35:24] <Reedy>	 I'm guessing it's config
[19:35:40] * James_F reads the static output.
[19:36:48] <bawolff>	 I wonder if somehow it reads a message in early initialization before the host is decided on, and then that gets cached for the request
[19:37:26] <bawolff>	 It definitely seems like it must be reading from wikidata. enwiki beta never had that as the message value for mediawiki:mainpage, and there was no vandalism at translatewiki
[19:37:44] <James_F>	 Possibly. But that's a very odd breakage.
[19:37:49] <bawolff>	 And it doesn't appear in cache/l10n (at least on the deployment-deploy01
[19:37:51] <James_F>	 wmf-config/config-cache/conf-labs-enwiki.json looks fine to me.
[19:39:59] <bawolff>	 although it could be in apc cache at this point
[19:40:17] <James_F>	 If I just restart all the Beta Cluster MW boxes that'd go away, right?
[19:40:27] <James_F>	 High tech solutions for high tech problems.
[19:40:53] <James_F>	 Or is it an on-disc cache rather than just RAM-backed?
[19:41:17] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10Aklapper) Also Thanks @JJMC89 as I missed some tasks (sorry for that)!  >>! In T247891#5977017, @MBinder_WMF wrote: > It used to be possible to plug this into advanced search, but I am struggling to figure that out n...
[19:43:36] <bawolff>	 James_F: That would clear the apc cache (which may or may not be related
[19:43:53] <James_F>	 Shall we give it a try?
[19:44:02] <bawolff>	 I'm pretty sure its not the ondisk cache, because i looked at all those
[19:44:09] <James_F>	 Right.
[19:44:18] <bawolff>	 sure, why not
[19:45:49] <bawolff>	 Still doesn't explain what went wrong in the first place though
[19:45:52] <James_F>	 !log Restarting deployment-prep's deployment-mediawiki-07 and deployment-mediawiki-09
[19:45:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:45:54] <James_F>	 No.
[19:46:25] <wikibugs>	 10Phabricator: Vandalism on Structured Data tasks - https://phabricator.wikimedia.org/T247891 (10MBinder_WMF) @Aklapper That's it! I could not find it because that field does not show unless the URL has been altered: https://phabricator.wikimedia.org/maniphest/query/advanced/
[19:47:40] <James_F>	 OK, well, they're back up and still exhibiting the behaviour.
[19:48:42] <James_F>	 !log Deleted deployment-mediawiki-parsoid10, no longer used.
[19:48:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:49:34] <James_F>	 !log Deleted deployment-dumps-puppetmaster02 for T241719
[19:49:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[19:49:36] <stashbot>	 T241719: Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3 - https://phabricator.wikimedia.org/T241719
[19:54:49] * bawolff tries some live debugging
[20:02:12] <bawolff>	 MessageCache::singleton()->get( 'mainpage', false ) [no db] returns correct main page, so that's something
[20:03:16] <James_F>	 Huh.
[20:05:17] <bawolff>	 so its definitely in the override code
[20:30:55] <bawolff>	 Ok, I think its because $wgMessacheCache has a keyspace of 'local' on both wikidata and enwiki, where it should be the wikiid
[20:31:12] <bawolff>	 Probably wikidata wins because it is alphabetically last when caching is being rebuild
[20:36:58] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:38:03] <wikibugs>	 10VPS-project-codesearch: codesearch is not searching package-lock.json - https://phabricator.wikimedia.org/T241033 (10Legoktm) I don't actually know what changed, but it appears to work for some extensions: https://codesearch.wmflabs.org/search/?q=minimist&i=nope&files=&repos=
[20:38:07] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:38:12] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:40:43] <legoktm>	 James_F, Krin<tab>: fyi that https://gerrit.wikimedia.org/r/q/hashtag:%22n%253Bminimist%253D1.2.5%22+(status:open%20OR%20status:merged) is moving us to grunt 1.1.0
[20:41:27] <James_F>	 Oh.
[20:41:44] <James_F>	 Should we change that to the standard?
[20:42:37] <James_F>	 https://gerrit.wikimedia.org/r/#/c/labs/libraryupgrader/config/+/580453
[20:44:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Cache, 10User-zeljkofilipin: Main pages of several Beta Cluster wikis redirect to other production wikis - https://phabricator.wikimedia.org/T247078 (10Bawolff) So it looks like what is happening:  * $wgMessageCacheType is set to CACHE_ACCEL on web  but CACHE_NONE f...
[20:45:41] <James_F>	 bawolff: Nice detective work.
[20:46:04] <bawolff>	 thanks
[20:46:16] <legoktm>	 James_F: yes, but IIRC libup behaves a bit weirdly if we upgrade it via both config and npm audit, so lets let the npm audit run go through and then merge config for any stragglers
[20:46:43] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Cache, 10User-zeljkofilipin: Main pages of several Beta Cluster wikis redirect to other production wikis (MessageCache keyspace is same for all wikis causing conflicts) - https://phabricator.wikimedia.org/T247078 (10Bawolff)
[20:46:47] <James_F>	 legoktm: WFM. See also https://gerrit.wikimedia.org/r/c/labs/libraryupgrader/config/+/574148
[20:48:06] <bawolff>	 Also, is beta just generally broken, or did my testing somehow cause everything to explode
[20:48:14] <bawolff>	 I don't think i did anything that could cause things to explode...
[20:48:25] <James_F>	 Err.
[20:48:28] <James_F>	 Explode?
[20:48:42] <James_F>	 It does indeed look down.
[20:50:27] <bawolff>	 pretty sure wasn't me
[20:52:47] <James_F>	 Servers are up.
[20:53:13] <James_F>	 Though wow is deployment-cache-text05.deployment-prep.eqiad.wmflabs out of date with puppet.
[20:53:45] <James_F>	 28 days out of sync.
[20:54:42] <wikibugs>	 (03PS1) 10QChris: Allow “Gerrit Managers” to import history [tools/cli] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/580463
[20:54:44] <wikibugs>	 (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [tools/cli] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/580463 (owner: 10QChris)
[20:54:48] <wikibugs>	 (03PS1) 10QChris: Import done. Revoke import grants [tools/cli] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/580464
[20:54:50] <wikibugs>	 (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [tools/cli] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/580464 (owner: 10QChris)
[20:56:57] <wikibugs>	 (03PS1) 10Ottomata: EventLogging now depends on EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/580466 (https://phabricator.wikimedia.org/T244521)
[20:57:30] <wikibugs>	 (03CR) 10Ottomata: "Jenkins failing on EventLogging change:" [integration/config] - 10https://gerrit.wikimedia.org/r/580466 (https://phabricator.wikimedia.org/T244521) (owner: 10Ottomata)
[20:57:41] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] EventLogging now depends on EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/580466 (https://phabricator.wikimedia.org/T244521) (owner: 10Ottomata)
[20:58:56] <wikibugs>	 (03PS2) 10Ottomata: EventLogging now depends on EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/580466 (https://phabricator.wikimedia.org/T244521)
[21:01:35] <wikibugs>	 (03CR) 10Mforns: [C: 03+1] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/580466 (https://phabricator.wikimedia.org/T244521) (owner: 10Ottomata)
[21:06:47] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 93055 bytes in 0.977 second response time
[21:07:56] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 93522 bytes in 1.022 second response time
[21:08:05] <shinken-wm>	 RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 52516 bytes in 1.084 second response time
[21:26:36] <James_F>	 bawolff: You fixed it (or it fixed itself).
[21:26:59] <bawolff>	 Yeah, wasn't me
[21:45:22] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] EventLogging now depends on EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/580466 (https://phabricator.wikimedia.org/T244521) (owner: 10Ottomata)
[21:46:14] <wikibugs>	 (03Merged) 10jenkins-bot: EventLogging now depends on EventStreamConfig [integration/config] - 10https://gerrit.wikimedia.org/r/580466 (https://phabricator.wikimedia.org/T244521) (owner: 10Ottomata)
[22:25:40] <wikibugs>	 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Krinkle) >>! In T99740#5941838, @ori wrote: >  > This might help: >  > `lang=p...
[22:40:48] <wikibugs>	 (03CR) 10Hashar: "I went to gather evidences for the MediaWiki train. I haven't tried this :/" [integration/config] - 10https://gerrit.wikimedia.org/r/578880 (owner: 10Hashar)
[22:49:25] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Deployment services), 10Scap, 10Operations, 10serviceops: Scap can't clear opcache on mw servers in Beta Cluster - https://phabricator.wikimedia.org/T237033 (10thcipriani) This one:  ` Job ['/usr/bin/scap', 'pull', '--no-php-restart', '--no-updat...
[23:18:46] <wikibugs>	 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Joe) This seems really excessive, especially if we ever want to run in a conta...