[00:00:45] (03PS1) 10Legoktm: Enable extension-coverage for OAuth [integration/config] - 10https://gerrit.wikimedia.org/r/449107 [00:01:21] (03CR) 10Legoktm: [C: 032] Enable extension-coverage for ParserFunctions [integration/config] - 10https://gerrit.wikimedia.org/r/449035 (owner: 10Legoktm) [00:01:23] (03CR) 10Legoktm: [C: 032] Enable extension-coverage for OAuth [integration/config] - 10https://gerrit.wikimedia.org/r/449107 (owner: 10Legoktm) [00:01:25] (03CR) 10Legoktm: [C: 032] Add myself (User:Ciencia Al Poder) to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/449087 (owner: 10Martineznovo) [00:02:28] (03Merged) 10jenkins-bot: Enable extension-coverage for ParserFunctions [integration/config] - 10https://gerrit.wikimedia.org/r/449035 (owner: 10Legoktm) [00:02:40] (03Merged) 10jenkins-bot: Enable extension-coverage for OAuth [integration/config] - 10https://gerrit.wikimedia.org/r/449107 (owner: 10Legoktm) [00:03:01] (03Merged) 10jenkins-bot: Add myself (User:Ciencia Al Poder) to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/449087 (owner: 10Martineznovo) [00:03:30] !log deploying https://gerrit.wikimedia.org/r/449035 https://gerrit.wikimedia.org/r/449107 https://gerrit.wikimedia.org/r/449087 [00:03:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:11:53] (03CR) 10Hashar: Move OAuthAuthentication to composer unittests (#2) (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/448858 (owner: 10Legoktm) [08:12:18] hashar: sorry, that's what I get for doing a direct revert without looking [08:12:53] legoktm: no worries :]]] [08:13:03] then I am not sure why it fails now [08:13:10] since it used to pass with vendor but that is not a big deal [08:13:24] in fall this year, I will probably migrate them all to use composer instead of vendor [08:14:12] (03PS1) 10Hashar: OAuthAuthentication back to Quibble with Composer [integration/config] - 10https://gerrit.wikimedia.org/r/449133 [08:14:28] legoktm: it is not like anyone will loose millions or someone life is at risk :-] [08:14:38] :P [08:14:54] and I am going to migrate your coverage-patch jobs to Docker images over today and tomorrow [08:15:08] then I will go on vacations [08:15:29] (03CR) 10Hashar: [C: 032] OAuthAuthentication back to Quibble with Composer [integration/config] - 10https://gerrit.wikimedia.org/r/449133 (owner: 10Hashar) [08:17:00] (03Merged) 10jenkins-bot: OAuthAuthentication back to Quibble with Composer [integration/config] - 10https://gerrit.wikimedia.org/r/449133 (owner: 10Hashar) [08:17:15] hashar: :) I'll be around to babysit as well [08:17:52] and I need to write down a blog post of what happened with Quibble since July 1st [08:17:54] bah busy busy [08:41:29] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Discovery-Search (Current work): Help needed to setup the elastic6 dev branch on CirrusSearch, Elastica and mediawiki-vendor - https://phabricator.wikimedia.org/T200669 (10dcausse) p:05Triage>03Normal [08:41:42] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Discovery-Search (Current work): Help needed to setup the elastic6 dev branch on CirrusSearch, Elastica and mediawiki-vendor - https://phabricator.wikimedia.org/T200669 (10dcausse) a:05dcausse>03None [08:48:52] 10Gerrit, 10Community-Tech, 10GitHub-Mirrors, 10MediaWiki-extensions-TemplateWizard: Mirror TemplateWizard on Github - https://phabricator.wikimedia.org/T200663 (10MarcoAurelio) [09:02:47] https://doc.wikimedia.org/cover-extensions/AntiSpoof-docker/ [09:02:49] * hashar flexes [09:08:47] niceee [09:09:03] hashar: looks identical [09:11:03] Is someone investigating why there are timeouts in https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/9089/console ? [09:12:14] hmm [09:12:24] WikibaseLexeme probably causing them? [09:12:56] it could be just a victim, I didn't check closely [09:14:01] 04:34:12 Unhandled rejection Error: failed-save: The save has failed. [09:14:34] but yeah looks like lexeme [09:22:54] (03PS4) 10Hashar: Extension coverage jobs on Docker [integration/config] - 10https://gerrit.wikimedia.org/r/448549 (https://phabricator.wikimedia.org/T195918) [09:24:49] (03CR) 10jerkins-bot: [V: 04-1] Extension coverage jobs on Docker [integration/config] - 10https://gerrit.wikimedia.org/r/448549 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [09:27:02] (03PS5) 10Hashar: Extension coverage jobs on Docker [integration/config] - 10https://gerrit.wikimedia.org/r/448549 (https://phabricator.wikimedia.org/T195918) [09:29:48] (03CR) 10Hashar: [C: 032] Extension coverage jobs on Docker [integration/config] - 10https://gerrit.wikimedia.org/r/448549 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [09:31:41] (03Merged) 10jenkins-bot: Extension coverage jobs on Docker [integration/config] - 10https://gerrit.wikimedia.org/r/448549 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [10:00:47] 10Phabricator, 10Regression: Creating subtasks sets more restrictive edit permissions than parent task: Others cannot edit such subtasks anymore - https://phabricator.wikimedia.org/T199122 (10MGChecker) I can confirm this is not only confusing but *really* annoying and disturbing in daily work on Phabricator.... [10:16:31] (03PS1) 10Matěj Suchánek: Fix documentation of ExtendClassUsageSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449143 [10:19:49] 10MediaWiki-Codesniffer: Look for @parma typo (should be @param) - https://phabricator.wikimedia.org/T182057 (10matej_suchanek) 05Open>03Resolved [10:26:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Move MediaWiki extension PHPUnit coverage jobs to docker + quibble - https://phabricator.wikimedia.org/T195918 (10hashar) [10:30:35] (03PS1) 10Hashar: Fix hardcoded sha1 in mwext-phpunit-coverage-patch.sh [integration/config] - 10https://gerrit.wikimedia.org/r/449147 (https://phabricator.wikimedia.org/T195918) [10:30:47] (03CR) 10Hashar: [C: 032] Fix hardcoded sha1 in mwext-phpunit-coverage-patch.sh [integration/config] - 10https://gerrit.wikimedia.org/r/449147 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [10:32:40] (03Merged) 10jenkins-bot: Fix hardcoded sha1 in mwext-phpunit-coverage-patch.sh [integration/config] - 10https://gerrit.wikimedia.org/r/449147 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [10:46:00] (03PS1) 10PleaseStand: Detect nesting of inline IF statements without parentheses [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449149 (https://phabricator.wikimedia.org/T171520) [10:46:46] (03PS1) 10Hashar: MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449150 (https://phabricator.wikimedia.org/T195918) [10:46:55] 10MediaWiki-Codesniffer: Add sniff to prevent using config globals when Config is available - https://phabricator.wikimedia.org/T200679 (10matej_suchanek) [10:47:03] (03CR) 10jerkins-bot: [V: 04-1] Detect nesting of inline IF statements without parentheses [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449149 (https://phabricator.wikimedia.org/T171520) (owner: 10PleaseStand) [10:49:03] (03CR) 10jerkins-bot: [V: 04-1] MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449150 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [10:50:33] (03PS2) 10Hashar: MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449150 (https://phabricator.wikimedia.org/T195918) [10:52:10] (03PS2) 10PleaseStand: Detect nesting of inline IF statements without parentheses [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449149 (https://phabricator.wikimedia.org/T171520) [10:53:06] (03CR) 10jerkins-bot: [V: 04-1] Detect nesting of inline IF statements without parentheses [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449149 (https://phabricator.wikimedia.org/T171520) (owner: 10PleaseStand) [11:02:00] (03PS3) 10PleaseStand: Detect nesting of inline IF statements without parentheses [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449149 (https://phabricator.wikimedia.org/T171520) [11:37:28] (03PS3) 10Hashar: MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449150 (https://phabricator.wikimedia.org/T195918) [11:46:47] (03PS1) 10Matěj Suchánek: Enable Squiz.PHP.NonExecutableCodeSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449159 (https://phabricator.wikimedia.org/T168465) [11:47:17] 10MediaWiki-Codesniffer, 10Patch-For-Review: add or create sniff to detect unreachable code after break in switch - https://phabricator.wikimedia.org/T168465 (10matej_suchanek) a:03matej_suchanek [11:47:26] (03CR) 10jerkins-bot: [V: 04-1] Enable Squiz.PHP.NonExecutableCodeSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449159 (https://phabricator.wikimedia.org/T168465) (owner: 10Matěj Suchánek) [11:58:01] (03PS2) 10Matěj Suchánek: Enable Squiz.PHP.NonExecutableCodeSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449159 (https://phabricator.wikimedia.org/T168465) [11:59:08] (03CR) 10jerkins-bot: [V: 04-1] Enable Squiz.PHP.NonExecutableCodeSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/449159 (https://phabricator.wikimedia.org/T168465) (owner: 10Matěj Suchánek) [12:01:05] 10Continuous-Integration-Infrastructure: Allow running individual tests on Jenkins - https://phabricator.wikimedia.org/T200684 (10Simetrical) [13:50:43] 10Continuous-Integration-Config, 10WikimediaMessages: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10Reedy) [13:54:02] 10Gerrit, 10Community-Tech, 10GitHub-Mirrors, 10MediaWiki-extensions-TemplateWizard: Mirror TemplateWizard on Github - https://phabricator.wikimedia.org/T200663 (10Reedy) Gotta create the empty repo on github first. Done at https://github.com/wikimedia/mediawiki-extensions-TemplateWizard Next push should... [13:57:44] (03CR) 10Hashar: [C: 032] MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449150 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [13:58:51] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Move MediaWiki extension PHPUnit coverage jobs to docker + quibble - https://phabricator.wikimedia.org/T195918 (10hashar) [13:59:44] (03Merged) 10jenkins-bot: MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449150 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [14:01:21] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811 (10Osnard) @hashar We have failing tests with "store-api-unittests" [1]. We have fixture data defined in a... [14:01:59] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Move MediaWiki extension PHPUnit coverage jobs to docker + quibble - https://phabricator.wikimedia.org/T195918 (10hashar) Left to migrate is the daily job `mediawiki-core-code-coverage` w... [14:12:54] 10Continuous-Integration-Config, 10WikimediaMessages: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10MarcoAurelio) What makes you think Jenkins do not work on WikimediaMessages? I see https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions/Wik... [14:16:22] 10Release-Engineering-Team, 10MediaWiki-extensions-WikimediaIncubator, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10StevenJ81) Going back to @Urbanecm 's comments above: With a handful of modifications, that's probably a reasonable way to start... [14:23:47] 10Gerrit, 10Community-Tech, 10GitHub-Mirrors, 10MediaWiki-extensions-TemplateWizard: Mirror TemplateWizard on Github - https://phabricator.wikimedia.org/T200663 (10MarcoAurelio) Now there: https://github.com/wikimedia/mediawiki-extensions-TemplateWizard If desired repository description like https://githu... [14:38:41] 10Continuous-Integration-Config, 10WikimediaMessages: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10dbarratt) >>! In T200691#4461588, @MarcoAurelio wrote: > What makes you think Jenkins do not work on WikimediaMessages? I see https://gerrit.wikimedia.org/r/#/q/s... [14:41:18] 10Continuous-Integration-Config, 10WikimediaMessages: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10dbarratt) But since this is the only test: https://phabricator.wikimedia.org/diffusion/EWME/browse/master/tests/phpunit/WikimediaMessagesTest.hooks.php then none... [14:50:42] 10Release-Engineering-Team, 10Structured-Data-Commons, 10Wikidata, 10User-Urbanecm, 10Wiki-Setup (Create): Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10daniel) [14:53:51] 10Release-Engineering-Team, 10Structured-Data-Commons, 10Wikidata, 10User-Urbanecm, 10Wiki-Setup (Create): Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10daniel) @Krinkle In my mind, the purpose of testwiki and friends is to have an ea... [14:56:59] 10Release-Engineering-Team, 10Structured-Data-Commons, 10Wikidata, 10User-Urbanecm, 10Wiki-Setup (Create): Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10daniel) >>! In T197616#4456669, @Fjalapeno wrote: > From what is being said, it s... [14:59:57] 10Continuous-Integration-Config, 10WikimediaMessages, 10Anti-Harassment (AHT Sprint 25), 10Patch-For-Review: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10dbarratt) [15:00:10] 10Continuous-Integration-Config, 10WikimediaMessages, 10Anti-Harassment (AHT Sprint 25), 10Patch-For-Review: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10dbarratt) a:03dbarratt [15:12:36] 10Continuous-Integration-Config, 10WikimediaMessages, 10Anti-Harassment (AHT Sprint 25), 10Patch-For-Review: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10dbarratt) @Legoktm That doesn't seem to have fixed it... is there something else that needs to change? [15:18:17] 10Continuous-Integration-Infrastructure: jenkins-bot is sometimes not added as CC - https://phabricator.wikimedia.org/T200702 (10Simetrical) [15:20:44] 10Continuous-Integration-Infrastructure: jenkins-bot is sometimes not added as CC - https://phabricator.wikimedia.org/T200702 (10Paladox) jenkins-bot does not need to be in cc to be able to run the tests. Jenkins will add it's self as a cc when it publishes a comment. [15:23:32] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Halfak) I'm OK with the move to wikimedia github, but gerrit would be really difficult for our more scholarly collaborat... [15:26:33] 10Continuous-Integration-Infrastructure: jenkins-bot is sometimes not added as CC - https://phabricator.wikimedia.org/T200702 (10Simetrical) 05Open>03Invalid Oh, good to know. Never mind, then. [15:39:27] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Ladsgroup) >>! In T194212#4461836, @Halfak wrote: > It seems like that was the driving need behind this move, right?... [16:11:51] (03PS1) 10Hashar: MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449239 (https://phabricator.wikimedia.org/T195918) [16:16:26] (03PS2) 10Hashar: MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449239 (https://phabricator.wikimedia.org/T195918) [16:18:11] (03CR) 10Hashar: [C: 032] MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449239 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [16:20:35] (03Merged) 10jenkins-bot: MediaWiki patch coverage job to Quibble/Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449239 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [16:27:14] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments, and 2 others: 1.32.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T191060 (10zeljkofilipin) 1.32.0-wmf.14 is deployed everywhere, all is good. [16:29:29] 10Beta-Cluster-Infrastructure, 10Maps: deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 (10Mholloway) For some reason, on deployment-maps04, it seems /var/lib/postgresql/9.6/main is being treated as the data directory rather than /srv/postgresql/9.6/main, even though... [16:32:22] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Move MediaWiki extension PHPUnit coverage jobs to docker + quibble - https://phabricator.wikimedia.org/T195918 (10greg) p:05Triage>03Normal [16:32:47] 10Beta-Cluster-Infrastructure, 10Maps: deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 (10Krenair) [16:33:13] 10Beta-Cluster-Infrastructure, 10Maps: deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 (10Krenair) That's probably a good thing to look into though be careful as /srv has it's own problems: `/dev/mapper/vd-second--local--disk 60G 49G 7.5G 87% /srv` [16:33:26] 10Beta-Cluster-Infrastructure, 10Maps: deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 (10Krenair) [16:37:34] 10Gerrit, 10Community-Tech, 10GitHub-Mirrors, 10MediaWiki-extensions-TemplateWizard: Mirror TemplateWizard on Github - https://phabricator.wikimedia.org/T200663 (10Niharika) 05Open>03Resolved a:03Niharika Thanks @Reedy! [16:39:46] thcipriani: can you take a look at this patchset for readers web when you get a chance? it enables jsdocs as an experimental job https://gerrit.wikimedia.org/r/#/c/integration/config/+/447454/ [16:39:56] PROBLEM - Host deployment-cassandra3-02 is DOWN: CRITICAL - Host Unreachable (10.68.21.237) [16:40:06] PROBLEM - Host deployment-cassandra3-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.103) [16:48:02] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:49:52] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Discovery-Search (Current work): Help needed to setup the elastic6 dev branch on CirrusSearch, Elastica and mediawiki-vendor - https://phabricator.wikimedia.org/T200669 (10hashar) a:03hashar Following releng team meeting, @dcausse lets... [16:57:45] PROBLEM - Puppet errors on deployment-elastic09 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [16:59:56] 10Release-Engineering-Team, 10Structured-Data-Commons, 10Wikidata, 10User-Urbanecm, 10Wiki-Setup (Create): Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10greg) >>! In T197616#4461721, @daniel wrote: > The reason to //also// do this on... [17:04:27] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:04:35] !log github: granted @thcipriani owner rights [17:04:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:08:42] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:09:19] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:09:27] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:11:54] Project mediawiki-core-code-coverage-docker build #1: 04FAILURE in 2 min 25 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/1/ [17:15:42] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [17:16:55] 10Release-Engineering-Team, 10Structured-Data-Commons, 10Wikidata, 10User-Urbanecm, 10Wiki-Setup (Create): Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10daniel) >>! In T197616#4462111, @greg wrote: > So, summary: given those limitatio... [17:28:07] 10Release-Engineering-Team, 10Structured-Data-Commons, 10Wikidata, 10User-Urbanecm, 10Wiki-Setup (Create): Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10greg) >>! In T197616#4462171, @daniel wrote: >>>! In T197616#4462111, @greg wrote... [17:35:43] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:50] (03PS1) 10Hashar: MediaWiki coverage job does not use castor [integration/config] - 10https://gerrit.wikimedia.org/r/449255 [17:37:05] Project mediawiki-core-code-coverage-docker build #2: 04STILL FAILING in 1 min 46 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/2/ [17:37:58] niedzielski: sorry for the delay, looking [17:39:16] (03CR) 10Hashar: [C: 032] MediaWiki coverage job does not use castor [integration/config] - 10https://gerrit.wikimedia.org/r/449255 (owner: 10Hashar) [17:41:41] (03Merged) 10jenkins-bot: MediaWiki coverage job does not use castor [integration/config] - 10https://gerrit.wikimedia.org/r/449255 (owner: 10Hashar) [17:43:42] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:45:35] Yippee, build fixed! [17:45:36] Project mediawiki-core-code-coverage-docker build #3: 09FIXED in 8 min 17 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/3/ [17:46:17] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Ladsgroup) @greg Hey, in our meeting @Halfak pointed out that we are storing large repos that is way beyond size limit o... [17:46:32] 10Release-Engineering-Team, 10Structured-Data-Commons, 10Wikidata, 10User-Urbanecm, 10Wiki-Setup (Create): Create a production test wiki in group0 to parallel Wikimedia Commons - https://phabricator.wikimedia.org/T197616 (10Krinkle) >>! In T197616#4461712, @daniel wrote: > Do you believe that the test wi... [17:47:09] 10Beta-Cluster-Infrastructure, 10Community-Tech, 10MediaWiki-extensions-LoginNotify, 10Wikimedia-Site-requests: Clarify LoginNotify emails are from beta - https://phabricator.wikimedia.org/T181908 (10Niharika) [17:47:22] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-LoginNotify, 10Wikimedia-Site-requests: Clarify LoginNotify emails are from beta - https://phabricator.wikimedia.org/T181908 (10Niharika) [17:49:19] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:50:16] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10greg) >>! In T194212#4462232, @Ladsgroup wrote: > @greg Hey, in our meeting @Halfak pointed out that we are storing larg... [17:51:32] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Services (doing), 10User-Eevans: Reset RESTBase deployment-prep environment - https://phabricator.wikimedia.org/T186750 (10mobrovac) a:03mobrovac [18:32:15] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [18:44:15] 10Continuous-Integration-Config, 10WikimediaMessages, 10Anti-Harassment (AHT Sprint 25), 10Patch-For-Review: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10MarcoAurelio) AFAIK extension registration autodetects PHPUnit tests inside the /tests/phpunit folder.... [18:49:51] 10Beta-Cluster-Infrastructure: Special:SpecialPages can not be viewed in beta cluster - https://phabricator.wikimedia.org/T200714 (10Bugreporter) [18:50:04] 10Beta-Cluster-Infrastructure: Special:SpecialPages can not be viewed in beta cluster - https://phabricator.wikimedia.org/T200714 (10Bugreporter) p:05Triage>03Unbreak! [19:01:56] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [19:10:20] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:11:43] (03CR) 10Hashar: [C: 031] Establish a new class for Zuul related things (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/447005 (owner: 10Dduvall) [19:13:27] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Services (done), 10User-Eevans: Reset RESTBase deployment-prep environment - https://phabricator.wikimedia.org/T186750 (10mobrovac) 05Open>03Resolved [19:13:36] greg-g, hey, about? [19:14:21] 10Beta-Cluster-Infrastructure, 10RESTBase, 10Services (done): RESTBase errors on logstash-beta - https://phabricator.wikimedia.org/T186994 (10mobrovac) 05Open>03Resolved a:03mobrovac Resolved via {T186750} [19:14:36] 10Beta-Cluster-Infrastructure, 10RESTBase, 10Services (done), 10User-Eevans: Reset RESTBase deployment-prep environment - https://phabricator.wikimedia.org/T186750 (10mobrovac) [19:15:01] (03PS1) 10Hashar: docker: drop filter from mediawiki-coverage.sh [integration/config] - 10https://gerrit.wikimedia.org/r/449272 (https://phabricator.wikimedia.org/T195918) [19:15:58] thcipriani, if there's a bug that may block group0 deployment but the branch has not yet been made, do you still list it as a subtask of the train blockers or do you wait? [19:16:07] 10Beta-Cluster-Infrastructure, 10Cassandra, 10RESTBase, 10Services (done): deployment-cassandra3-0{1,2}: Contact point 0 () is not a valid host name, the following values are valid contact points: ipAddress, hostName or ipAddress:port - https://phabricator.wikimedia.org/T192539 (10mobrovac) 05Open>03Res... [19:17:00] (03PS1) 10Hashar: Migrate mw coverage job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449273 (https://phabricator.wikimedia.org/T195918) [19:17:16] (03CR) 10Hashar: [C: 032] docker: drop filter from mediawiki-coverage.sh [integration/config] - 10https://gerrit.wikimedia.org/r/449272 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [19:17:59] 10Beta-Cluster-Infrastructure, 10Services (done): "invalid locale" warning on deployment-restbase02.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T195709 (10mobrovac) 05Open>03Resolved a:03mobrovac Not happening any longer after fixing {T186750}. [19:18:41] (03Merged) 10jenkins-bot: docker: drop filter from mediawiki-coverage.sh [integration/config] - 10https://gerrit.wikimedia.org/r/449272 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [19:18:50] mobrovac, did you forget to puppet node clean/deactivate on those cassandra3 hosts? [19:19:40] before i deleted them you mean Krenair? indeed i did :/ [19:19:50] well [19:19:52] *i did forget [19:19:59] doesn't have to be before you delete them [19:20:30] ah on the puppetmaster you mean [19:20:32] k [19:20:33] in fact the docs say to do it afterwards [19:21:54] done now Krenair, thnx for the reminder [19:21:58] np [19:23:27] Krenair: yes to your question to tyler, might as well track it/be aware [19:23:39] also, I'm just heading out for lunch duties, leave a message :) [19:23:44] k [19:24:29] btw I'm not 100% about my one but I reckon it might qualify as it's a fairly visible page that's broken, being linked from literally every page on the wiki [19:24:53] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T191061 (10Krenair) [19:33:43] Krenair: sorry, was in a meeting, I'd go ahead and add it to the train blocker task as greg-.g said [19:34:20] !log copied /etc/pam.d/sshd from deployment-restbase01 to deployment-restbase02 to fix missing MOTD, T195709 [19:34:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:34:24] T195709: "invalid locale" warning on deployment-restbase02.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T195709 [19:42:16] 10Beta-Cluster-Infrastructure: Request for shell access and steward rights in beta cluster - https://phabricator.wikimedia.org/T194267 (10Krenair) Yep. This isn't your account, is it? https://deployment.wikimedia.beta.wmflabs.org/wiki/Special:CentralAuth/Rxy It was registered the day after you opened this task,... [19:44:07] !log renaming Jenkins job "mediawiki-core-code-coverage" to "mediawiki-core-code-coverage-docker" to keep history. It is migrated by https://gerrit.wikimedia.org/r/#/c/integration/config/+/449273/ | T195918 [19:44:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:44:12] T195918: Move MediaWiki extension PHPUnit coverage jobs to docker + quibble - https://phabricator.wikimedia.org/T195918 [19:45:56] 10Beta-Cluster-Infrastructure: Request for shell access and steward rights in beta cluster - https://phabricator.wikimedia.org/T194267 (10MarcoAurelio) IIRC the global account deletion was to unblock an stuck global rename that failed in the middle of the process and left attached and unnatached accounts. I thin... [19:46:11] 10Beta-Cluster-Infrastructure: Request for shell access and steward rights in beta cluster - https://phabricator.wikimedia.org/T194267 (10Krenair) shell access done [19:46:13] Hauskatze, alright so that's definitely Rxy then? [19:46:33] I'd prefer if rxy could answer himself [19:46:55] I think he is though [19:46:55] ok [19:47:19] 10Beta-Cluster-Infrastructure: Request for shell access and steward rights in beta cluster - https://phabricator.wikimedia.org/T194267 (10Krenair) a:03Rxy [19:48:40] (03CR) 10Hashar: [C: 04-1] "Hold until https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/ pass properly" [integration/config] - 10https://gerrit.wikimedia.org/r/449273 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [19:49:46] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [19:49:58] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Move MediaWiki extension PHPUnit coverage jobs to docker + quibble - https://phabricator.wikimedia.org/T195918 (10hashar) The new mediawiki/core coverage job is being build at https://int... [19:50:20] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:50:51] 10Beta-Cluster-Infrastructure: deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10Krenair) CCing some people who have touched this submodule before [19:52:20] 10Beta-Cluster-Infrastructure: deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10MarcoAurelio) I am not aware of having voluntarily touched that config before. Perhaps the 'scap' tests? Or maintenance scripts? [19:56:09] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External): Beta cluster scap job ( beta-scap-eqiad ) fails due to puppet erasing /etc/ssh/ssh_known_hosts - https://phabricator.wikimedia.org/T159332 (10Krenair) 05stalled>03Resolved a:03Krenair When I set up puppet DB again in beta th... [19:56:58] 10Beta-Cluster-Infrastructure: deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10Krenair) >>! In T199270#4462667, @MarcoAurelio wrote: > I am not aware of having voluntarily touched that config before. https://gerrit.wikimedia.org/r/#/c/... [19:59:02] 10Beta-Cluster-Infrastructure: deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10EBernhardson) For the search part of this, we do reference the schemas in there at runtime to encode some logging. That schema hasn't changed in some time (y... [19:59:29] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Technical-Debt, 10Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220 (10Krenair) [19:59:42] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Technical-Debt, 10Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220 (10Krenair) [20:02:44] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560 (10Krenair) Has anyone seen this or T186133 lately? [20:02:57] 10Beta-Cluster-Infrastructure: deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10MarcoAurelio) https://phabricator.wikimedia.org/source/mediawiki-config/history/master/wmf-config/event-schemas/ Apparently this is something a team manages... [20:07:45] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Create crhwiki and srwiki on beta - https://phabricator.wikimedia.org/T196583 (10Krenair) Did this get done? [20:09:50] 10Beta-Cluster-Infrastructure, 10Mathoid: Move mathoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142255 (10Krenair) [20:09:53] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Technical-Debt, 10Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220 (10Krenair) [20:09:55] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Performance-Team, 10Performance: Using fully-qualified function calls is faster - https://phabricator.wikimedia.org/T200629 (10aaron) p:05Triage>03Low [20:12:50] thcipriani, where are we with deployment-deploy01? [20:13:06] I looked at it Friday, it looks ready to go to me. [20:13:28] cool so we just get someone with integration privileges to swap jenkins to using the new host? [20:14:03] yep, we'll need to make it the deploy master so that it's unlocked for deployments [20:14:46] I can add it to jenkins and re-point the labels appropriately, that's on my todo list for this week. [20:15:19] uh oh, old stuff detected: [20:15:22] jjb/beta.yaml:# Jobs should be prefixed with 'beta-' and tied to the 'deployment-bastion' [20:15:22] jjb/beta.yaml:# Basic valid defaults, making sure we run on the deployment-bastion host [20:15:29] in integration/config.git [20:15:34] luckily just comments [20:15:47] actually were you here when we had deployment-bastion? [20:16:05] I think that pre-dates me [20:17:46] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Jenkins, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 (10Krenair) thcipriani, where are we with deployment-deploy01? I looked at it Fr... [20:18:53] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Move MediaWiki extension PHPUnit coverage jobs to docker + quibble - https://phabricator.wikimedia.org/T195918 (10hashar) [20:19:17] (03PS1) 10Alex Monk: Update references to deployment-bastion [integration/config] - 10https://gerrit.wikimedia.org/r/449288 [20:23:40] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Ladsgroup) I know, we were thinking of the middle step (moving from wiki-ai organization to wikimedia origination in git... [20:26:10] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Halfak) FWIW, getting LFS in gerrit/phab was critical for our deployment processes and regardless of where development i... [20:30:46] 10Beta-Cluster-Infrastructure, 10Maps, 10Maps-Sprint, 10Reading-Infrastructure-Team-Backlog (Kanban): deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 (10Mholloway) [20:31:03] 10Beta-Cluster-Infrastructure, 10Maps, 10Maps-Sprint, 10Reading-Infrastructure-Team-Backlog (Kanban): deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 (10Mholloway) a:03Mholloway [20:31:29] 10Beta-Cluster-Infrastructure, 10Maps, 10Maps-Sprint, 10Reading-Infrastructure-Team-Backlog (Kanban): deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 (10Mholloway) p:05Triage>03High [20:35:55] 10Beta-Cluster-Infrastructure: deployment-tin:/srv/mediawiki-staging/wmf-config/event-schemas is out of sync - https://phabricator.wikimedia.org/T199270 (10EBernhardson) That wasn't reverted because it was the wrong one, it was reverted because bumping that submodule was completely unrelated to the patch being d... [20:57:58] !log deployment-maps04 restarted postgresql to pick up the correct data_directory value (T200634) [20:57:59] see this https://groups.google.com/forum/#!topic/repo-discuss/H4pgIYhgEc4 to provide feedback on the change view in polygerrit [20:58:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:58:02] T200634: deployment-maps04 root partition space warnings - https://phabricator.wikimedia.org/T200634 [21:01:17] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Create crhwiki and srwiki on beta - https://phabricator.wikimedia.org/T196583 (10Reedy) 05Open>03Resolved a:03Reedy >>! In T196583#4263901, @gerritbot wrote: > Change 437983 **merged** by jenkins-bot: > [operations/mediawiki-config@master] Add some new b... [21:07:34] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: logstash-beta.wmflabs.org default dashboard missing - https://phabricator.wikimedia.org/T184602 (10Krenair) >>! In T184602#3889630, @Paladox wrote: > Does this use mediawiki vagrant for logstash installation? I'm pretty sure nothing in beta is/should be usi... [21:09:55] tgr, hey [21:10:10] Do you know if there's anything left for https://phabricator.wikimedia.org/T84950 ? [21:13:34] Krenair: I haven't followed closely [21:13:55] at a glance, there doesn't seem to be a thumbor box in deployment-prep [21:14:16] there is [21:14:21] deployment-imagescaler0* [21:14:35] well [21:14:38] at least 01 is running thumbor [21:14:50] 02 has haproxy for thumbor, but... [21:16:03] ah, ok [21:16:04] for some reason it's not running thumbor itself. hm [21:16:14] I'm not aware of any other difference [21:16:45] but as I said, I haven't looked at it closely since you killed the local thumb.php hacks [21:17:15] ok [21:17:58] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet, 10Technical-Debt, 10Tracking: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220 (10Krenair) [21:18:01] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Multimedia, 10Patch-For-Review: Images with extra parameters (e.g. low-quality JPEG) are not rendered in beta - https://phabricator.wikimedia.org/T71757 (10Krenair) [21:18:06] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Multimedia, 10Tracking: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950 (10Krenair) 05Open>03Resolved a:03Krenair I think betwee... [21:22:19] 10Beta-Cluster-Infrastructure: Create crhwiki and srwiki on beta - https://phabricator.wikimedia.org/T196583 (10Petar.petkovic) [21:24:38] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Multimedia, 10Tracking: Thumbnail generation should happen via the same setup in the beta cluster and in production (tracking) - https://phabricator.wikimedia.org/T84950 (10Krenair) [21:24:42] 10Beta-Cluster-Infrastructure: beta cluster missing vips command needed to render tiffs and pngs, pnmtojpeg for DjVus - https://phabricator.wikimedia.org/T116816 (10Krenair) 05Open>03Resolved a:03Krenair I suspect {T199938} took care of this. [21:25:46] 10Continuous-Integration-Infrastructure, 10Quibble: Allow running individual tests on Jenkins - https://phabricator.wikimedia.org/T200684 (10hashar) We have defined [[ https://www.mediawiki.org/wiki/Continuous_integration/Entry_points | a few test entries points ]] and one should be able to run the linter loca... [21:34:47] (03PS2) 10Hashar: Migrate mw coverage job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449273 (https://phabricator.wikimedia.org/T195918) [21:35:17] Project mediawiki-core-code-coverage-docker build #3665: 15ABORTED in 2 min 14 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/3665/ [21:37:28] (03PS3) 10Hashar: Migrate mw coverage job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/449273 (https://phabricator.wikimedia.org/T195918) [21:37:58] (03CR) 10Hashar: [C: 04-2] "Bumped timeout to 4 hours. Switched to vendor + sqlite." [integration/config] - 10https://gerrit.wikimedia.org/r/449273 (https://phabricator.wikimedia.org/T195918) (owner: 10Hashar) [21:38:14] Project mediawiki-core-code-coverage-docker build #3664: 15ABORTED in 1 hr 49 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/3664/ [21:38:27] !log ran package upgrades on deployment-imagescaler02 to fix thumbor [21:38:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:41:01] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10greg) >>! In T194212#4462790, @Halfak wrote: > FWIW, getting LFS in gerrit/phab was critical for our deployment processe... [21:41:29] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10greg) (for the record, that's how git-annex works, which is my mental model for how all these things should work ;) ) [21:56:43] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [22:01:12] 10Gerrit, 10Community-Tech, 10GitHub-Mirrors, 10MediaWiki-extensions-TemplateWizard: Mirror TemplateWizard on Github - https://phabricator.wikimedia.org/T200663 (10Samwilson) 05Resolved>03Open Thanks! I think a repo description would be good, but with the project website being https://www.mediawiki.or... [22:11:18] Project mediawiki-core-code-coverage-docker build #3666: 15ABORTED in 31 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/3666/ [22:17:32] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Move Blubber repo and CR to Gerrit - https://phabricator.wikimedia.org/T200452 (10dduvall) An [[ https://gerrit-review.googlesource.com/c/plugins/go-import/+/190630 | upstream patch ]] has been submitted to fix the auth requirement for the `go-... [23:03:41] !log Delete and recreate deployment-webperf13 (T195312 / T180761) [23:03:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:03:46] T180761: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 [23:03:47] T195312: Move flame graphs hosting from mwlog1001 to webperf-2 and enable in Beta Cluster - https://phabricator.wikimedia.org/T195312 [23:04:52] !log Create instance deployment-webperf13 (deployment-webperf13 ) - T195312 / T180761 [23:04:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:06:41] !log Perform node deactivate/clean for deleted deployment-webperf13 [23:06:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:06:45] PROBLEM - Host deployment-webperf13 is DOWN: CRITICAL - Host Unreachable (10.68.22.102) [23:20:05] 10Continuous-Integration-Config, 10WikimediaMessages, 10Anti-Harassment (AHT Sprint 25), 10Patch-For-Review: Get WikimediaMessages' tests running on Jenkins - https://phabricator.wikimedia.org/T200691 (10Legoktm) >>! In T200691#4461788, @dbarratt wrote: > @Legoktm That doesn't seem to have fixed it... is t... [23:22:50] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1344 bytes in 0.011 second response time [23:25:54] that's weird [23:25:59] it's looking fine to me [23:26:55] PROBLEM - Puppet errors on deployment-webperf12 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:27:53] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 47296 bytes in 3.040 second response time [23:28:19] !log Setting up puppet cert for deployment-webperf12; T195312 / T180761 [23:28:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:28:24] T180761: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 [23:28:25] T195312: Move flame graphs hosting from mwlog1001 to webperf-2 and enable in Beta Cluster - https://phabricator.wikimedia.org/T195312 [23:31:54] RECOVERY - Puppet errors on deployment-webperf12 is OK: OK: Less than 1.00% above the threshold [0.0] [23:32:12] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T191061 (10Krenair) [23:38:27] 10Gerrit: Rename Gerrit Repo - https://phabricator.wikimedia.org/T200736 (10Paladox) [23:48:49] 10Gerrit: Place holder task for 2.16 upgrade - https://phabricator.wikimedia.org/T200739 (10Paladox) [23:48:51] 10Gerrit: Place holder task for 2.16 upgrade - https://phabricator.wikimedia.org/T200739 (10Paladox) p:05Triage>03Lowest