[01:51:20] RECOVERY - Free space - all mounts on deployment-kafka-jumbo-1 is OK: OK: All targets OK [02:03:40] RECOVERY - Free space - all mounts on deployment-kafka-jumbo-2 is OK: OK: All targets OK [02:15:49] PROBLEM - Host deployment-redis02 is DOWN: CRITICAL - Host Unreachable (10.68.16.231) [02:15:51] PROBLEM - Host deployment-redis01 is DOWN: CRITICAL - Host Unreachable (10.68.16.177) [02:18:02] PROBLEM - Host deployment-dumps-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.21.153) [02:24:20] PROBLEM - Host deployment-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (10.68.21.200) [03:13:23] 10Phabricator: Remove approval requirement for new accounts, or patch everything in Phabricator to allow unapproved users to be treated as logged out for permissions purposes - https://phabricator.wikimedia.org/T197550#4295677 (10Yair_rand) Can we set up a temporary bug tracker that users can be redirected to un... [05:23:54] 10Phabricator: Remove approval requirement for new accounts, or patch everything in Phabricator to allow unapproved users to be treated as logged out for permissions purposes - https://phabricator.wikimedia.org/T197550#4310180 (10Nemo_bis) p:05Triage>03High The reason why this is high priority is https://lis... [06:18:39] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4310242 (10Osnard) [06:33:40] 10Deployments, 10Release-Engineering-Team (Watching / External), 10Operations, 10HHVM, and 3 others: Translation cache exhaustion caused by changes to PHP code in file scope - https://phabricator.wikimedia.org/T103886#4310251 (10Joe) >>! In T103886#4307439, @MoritzMuehlenhoff wrote: > Does that actually st... [07:47:09] (03CR) 10Hashar: [V: 032 C: 032] Mark repository as read only [extensions/CommunityVoice] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/441361 (https://phabricator.wikimedia.org/T196618) (owner: 10MarcoAurelio) [07:47:48] (03CR) 10Hashar: [C: 032] Archive the CommunityVoice extension [integration/config] - 10https://gerrit.wikimedia.org/r/441358 (https://phabricator.wikimedia.org/T196618) (owner: 10MarcoAurelio) [07:49:24] (03Merged) 10jenkins-bot: Archive the CommunityVoice extension [integration/config] - 10https://gerrit.wikimedia.org/r/441358 (https://phabricator.wikimedia.org/T196618) (owner: 10MarcoAurelio) [07:49:44] !log github: deleting archived repository https://github.com/wikimedia/mediawiki-extensions-CommunityVoice | T196618 [07:49:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:49:47] T196618: Archive the CommunityVoice extension - https://phabricator.wikimedia.org/T196618 [08:08:00] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#1419799 (10Ladsgroup) I... [08:15:51] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4310502 (10jcrespo) Can... [08:36:53] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4310554 (10Ladsgroup) Ra... [08:49:23] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#1496870 (10Marostegui) K... [09:41:19] 10Phabricator, 10Patch-For-Review: Remove approval requirement for new accounts, or patch everything in Phabricator to allow unapproved users to be treated as logged out for permissions purposes - https://phabricator.wikimedia.org/T197550#4310752 (10Aklapper) >>! In T197550#4310132, @Yair_rand wrote: > Can we... [09:50:39] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automatize the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4310785 (10Ladsgroup) >>... [10:11:40] 10Phabricator, 10Release-Engineering-Team (Kanban), 10media-storage, 10Patch-For-Review: Connect Phabricator to swift for storage of git-lfs and file uploads. - https://phabricator.wikimedia.org/T182085#4310866 (10fgiunchedi) I think we're ready to try swift for phabricator in eqiad. I'm not opposed to try... [10:23:39] (03PS1) 10Hashar: Migrate WikibaseQuality to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441812 (https://phabricator.wikimedia.org/T183512) [10:24:52] (03PS1) 10Hashar: Migrate WikibaseQualityConstraints to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441813 (https://phabricator.wikimedia.org/T183512) [10:27:12] (03PS1) 10Hashar: Migrate WikibaseQualityExternalValidation to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441814 (https://phabricator.wikimedia.org/T183512) [10:29:04] (03PS1) 10Hashar: Migrate WikidataPageBanner to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441815 (https://phabricator.wikimedia.org/T183512) [10:30:20] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4310949 (10hashar) [10:30:35] (03CR) 10Hashar: [C: 032] Migrate WikibaseQuality to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441812 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:30:40] (03CR) 10Hashar: [C: 032] Migrate WikibaseQualityConstraints to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441813 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:30:43] (03CR) 10Hashar: [C: 032] Migrate WikibaseQualityExternalValidation to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441814 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:30:48] (03CR) 10Hashar: [C: 032] Migrate WikidataPageBanner to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441815 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:32:23] (03Merged) 10jenkins-bot: Migrate WikibaseQuality to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441812 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:32:26] (03Merged) 10jenkins-bot: Migrate WikibaseQualityConstraints to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441813 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:32:30] (03Merged) 10jenkins-bot: Migrate WikibaseQualityExternalValidation to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441814 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:32:32] (03Merged) 10jenkins-bot: Migrate WikidataPageBanner to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441815 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [10:47:24] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata.org: [Wikidata.org] AutoLoaderStructureTest::testPSR4Completeness fails - https://phabricator.wikimedia.org/T198077#4311008 (10hashar) p:05Triage>03Normal [10:51:14] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Configuration, 10Wikidata, 10Wikidata.org: [Wikidata.org] AutoLoaderStructureTest::testPSR4Completeness fails - https://phabricator.wikimedia.org/T198077#4311033 (10hashar) [11:07:11] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Configuration, 10Wikidata, 10Wikidata.org: [Wikidata.org] AutoLoaderStructureTest::testPSR4Completeness fails - https://phabricator.wikimedia.org/T198077#4311008 (10Legoktm) This *might* be an issue with the test itself, I'm not sure how it ha... [11:19:01] (03PS1) 10Zfilipin: Job running WikibaseLexeme Selenium tests daily [integration/config] - 10https://gerrit.wikimedia.org/r/441827 (https://phabricator.wikimedia.org/T194252) [11:27:47] (03Abandoned) 10Zfilipin: WIP Added WikibaseLexeme project for a daily node selenium test run against beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/434025 (https://phabricator.wikimedia.org/T194252) (owner: 10WMDE-leszek) [11:28:56] (03PS2) 10Zfilipin: Job running WikibaseLexeme Selenium tests daily [integration/config] - 10https://gerrit.wikimedia.org/r/441827 (https://phabricator.wikimedia.org/T194252) [11:30:00] 10Release-Engineering-Team (Kanban), 10Wikidata, 10Patch-For-Review, 10User-zeljkofilipin: jenkins_jobs.errors.JenkinsJobsException: Duplicate entry found in '/src/integration/config/jjb/selenium.yaml: 'WikibaseLexeme' already defined - https://phabricator.wikimedia.org/T197882#4311243 (10zeljkofilipin) 0... [11:30:44] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Configuration, 10Wikidata, 10Wikidata.org: [Wikidata.org] AutoLoaderStructureTest::testPSR4Completeness fails - https://phabricator.wikimedia.org/T198077#4311246 (10hashar) Extension registry does not normalizes the path at: ``` protected... [11:41:04] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4311256 (10hashar) [11:46:26] hashar: :D [11:46:35] legoktm: :]]] [11:46:39] legoktm: you should sleep!! [11:46:51] hashar: I'm in Europe! [11:47:00] \o/ [11:47:21] hashar: https://cloud.legoktm.com/index.php/s/SiNEwmknGsriEEy my view earlier today [11:48:22] make sure to climb/walk up there. There is a nice view [11:48:39] (unless there is pollution / smog, but based on the picture the sky looks all clear) [11:48:42] definitely, I think we're going to on Wednesday [11:56:14] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#4311316 (10mark) [11:58:52] 10Release-Engineering-Team (Kanban), 10User-greg, 10Wikimedia-extension-review-queue: Re-think/factor [[mw:Review queue]] and generally the process of getting new code into production - https://phabricator.wikimedia.org/T195244#4220553 (10Aklapper) [12:19:32] !log "Beta: Update cxserver to ece5e7a" [12:19:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:21:06] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4311490 (10hashar) [12:33:12] legoktm: my patch is broken eventually :( [12:33:19] legoktm: that fails on BlueSpiceFoundation bah [12:42:24] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4311556 (10hashar) [12:42:36] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4259250 (10hashar) [12:49:51] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Configuration, 10Wikidata, 10Wikidata.org, 10Patch-For-Review: [Wikidata.org] AutoLoaderStructureTest::testPSR4Completeness fails - https://phabricator.wikimedia.org/T198077#4311580 (10hashar) I went with a workaround in the extension : http... [13:26:50] addshore: let me know if you need anything for T197868 [13:26:50] T197868: `Class 'Wikibase\DataModel\Entity\ItemId' not found` error when trying to install WikibaseLexeme - https://phabricator.wikimedia.org/T197868 [13:27:05] I can copy/paste the entire terminal output, any file... [13:33:40] 10Release-Engineering-Team (Watching / External), 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, and 2 others: Automate the check and fix of object, schema and data drifts between mediawiki HEAD, production masters and slaves - https://phabricator.wikimedia.org/T104459#1548414 (10Marostegui) [13:38:14] 10Project-Admins, 10Developer-Relations: Sort out scope/confusion between #Possible-Tech-Projects and #Outreach-Programs-Projects tags - https://phabricator.wikimedia.org/T198101#4311807 (10Aklapper) [13:38:30] 10Project-Admins, 10Developer-Relations: Sort out scope between #MediaWiki-extension-requests vs. #Technical-Tool-Request tags - https://phabricator.wikimedia.org/T198102#4311817 (10Aklapper) [13:38:36] 10Project-Admins, 10Developer-Relations: Sort out scope/confusion between #Possible-Tech-Projects and #Outreach-Programs-Projects tags - https://phabricator.wikimedia.org/T198101#4311807 (10Aklapper) [14:05:44] hashar hi, the translate ext test for REL1_31 is failing, see https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm-jessie/50503/console [14:05:48] the error is "The module 'ext.uls.mediawiki' required by 'ext.translate.pagetranslation.uls' must exist" [14:07:13] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<11.11%) [14:08:38] paladox: fill a task please. Probably it needs backport from master [14:08:45] ok [14:09:46] paladox: or it lacks a dependency somehow :] [14:09:59] hashar timemediahandler also fails too [14:10:32] https://phabricator.wikimedia.org/T197933 [14:10:52] paladox: for Translate on https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm-jessie/50503/parameters/ , the EXT_DEPENDENCIES parameter does not have UniversalLanguageSelector being added [14:10:56] 10Continuous-Integration-Config, 10MediaWiki-extensions-Translate: Translate mw extension tests failing for branch REL1_31 - https://phabricator.wikimedia.org/T198110#4311999 (10Paladox) [14:11:00] ah [14:11:33] (03PS3) 10Zfilipin: WIP Job running WikibaseLexeme Selenium tests daily [integration/config] - 10https://gerrit.wikimedia.org/r/441827 (https://phabricator.wikimedia.org/T194252) [14:11:34] 'Translate': ['UniversalLanguageSelector', 'EventLogging', 'cldr'], [14:11:45] hashar though it's listed here https://github.com/wikimedia/integration-config/blob/master/zuul/parameter_functions.py#L319 [14:11:50] some how that is not being injected [14:11:58] as it is happening to timedmediahandler too [14:12:00] (deps) [14:12:13] will take a look at it later tonight if time allow [14:12:24] (03CR) 10jerkins-bot: [V: 04-1] WIP Job running WikibaseLexeme Selenium tests daily [integration/config] - 10https://gerrit.wikimedia.org/r/441827 (https://phabricator.wikimedia.org/T194252) (owner: 10Zfilipin) [14:13:00] ok thanks [14:27:35] (03PS4) 10Zfilipin: Job running WikibaseLexeme Selenium tests daily [integration/config] - 10https://gerrit.wikimedia.org/r/441827 (https://phabricator.wikimedia.org/T194252) [14:28:33] (03CR) 10Zfilipin: "PS4 reverts to PS2." [integration/config] - 10https://gerrit.wikimedia.org/r/441827 (https://phabricator.wikimedia.org/T194252) (owner: 10Zfilipin) [14:54:24] (03PS1) 10Elukey: Remove puppet submodules merged into operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/441879 (https://phabricator.wikimedia.org/T188377) [15:23:04] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Should selenium-EXTENSION-jessie run for all repositores with Selenium tests? - https://phabricator.wikimedia.org/T188742#4312322 (10zeljkofilipin) a:03zeljkofilipin [15:30:40] (03PS1) 10Zfilipin: Job running Echo Selenium tests daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/441892 (https://phabricator.wikimedia.org/T188742) [15:31:55] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Should selenium-EXTENSION-jessie run for all repositores with Selenium tests? - https://phabricator.wikimedia.org/T188742#4312345 (10zeljkofilipin) [15:44:13] (03CR) 10Zfilipin: [C: 032] Job running Echo Selenium tests daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/441892 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [15:46:27] (03Merged) 10jenkins-bot: Job running Echo Selenium tests daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/441892 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [16:02:01] twentyafterfour: wnt me to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/441806? [16:03:20] chasemp: sure [16:04:09] twentyafterfour hi, this https://phabricator.wikimedia.org/source/mediawiki/manage/ shows "Pull of 'mediawiki' failed: Command failed with error #128! COMMAND git fetch origin '+refs/*:refs/*' --prune STDOUT (empty) STDERR error: insufficient permission for adding an object to repository database objects fatal: failed to write object fatal: unpack-objects failed" [16:05:41] Project beta-scap-eqiad build #213338: 04FAILURE in 1 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213338/ [16:05:48] greg-g: paladox fixed I think [16:05:58] err sorry greg-g didn't mean to ping you [16:06:17] twentyafterfour thanks :) [16:08:49] twentyafterfour: yaml issues so I'm reverting [16:09:36] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:10:06] 10Phabricator, 10Patch-For-Review: Remove approval requirement for new accounts, or patch everything in Phabricator to allow unapproved users to be treated as logged out for permissions purposes - https://phabricator.wikimedia.org/T197550#4295677 (10chasemp) >>! In T197550#4310714, @gerritbot wrote: > Change 4... [16:13:59] Project beta-scap-eqiad build #213339: 04STILL FAILING in 7.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213339/ [16:14:36] 16:13:58 sudo: a password is required [16:14:43] well that's new [16:14:46] chasemp: hmm ok I'll test locally [16:14:52] paladox: hmm? [16:15:01] where do you see that? [16:15:02] twentyafterfour for scap above [16:15:03] twentyafterfour: ack thanks, I would I just don't have time atm [16:15:08] twentyafterfour https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213339/console [16:15:58] hmm weird [16:16:07] ldap went unreacheable I guess [16:16:18] I had the issue a few minuites ago as well [16:16:53] oh [16:17:32] rebuilding at https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213340/console [16:20:23] !log deployment-prep: git gc on a few repositories under /srv/mediawiki-staging/php-master [16:20:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:22:13] paladox: yeah it is running fine now [16:22:19] Yippee, build fixed! [16:22:19] Project beta-scap-eqiad build #213340: 09FIXED in 5 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/213340/ [16:22:20] hashar thanks :) [16:22:29] I have done nothing though [16:24:10] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Should selenium-EXTENSION-jessie run for all repositores with Selenium tests? - https://phabricator.wikimedia.org/T188742#4312478 (10zeljkofilipin) [16:27:00] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Should selenium-EXTENSION-jessie run for all repositores with Selenium tests? - https://phabricator.wikimedia.org/T188742#4312495 (10zeljkofilipin) [16:35:54] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4312528 (10hashar) [16:39:05] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Should selenium-EXTENSION-jessie run for all repositores with Selenium tests? - https://phabricator.wikimedia.org/T188742#4312546 (10zeljkofilipin) a:05zeljkofilipin>03None [16:40:27] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4312564 (10hashar) [16:40:44] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4260271 (10hashar) [16:41:29] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Establish shared library for pipeline code used in Jenkins - https://phabricator.wikimedia.org/T196940#4312573 (10dduvall) a:03dduvall [16:53:32] 10Release-Engineering-Team (Kanban), 10Scap: Document scap swat command - https://phabricator.wikimedia.org/T196411#4312651 (10mmodell) I haven't had a chance to work on this due to unplanned phabricator incident response. [16:56:11] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Scap canary has a shifting baseline - https://phabricator.wikimedia.org/T183999#4312660 (10thcipriani) a:05thcipriani>03None [16:56:51] 10Phabricator, 10Release-Engineering-Team (Kanban), 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#4312662 (10mmodell) This is currently not a priority due to other urgent... [16:57:08] 10Phabricator, 10Release-Engineering-Team (Kanban), 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#4312663 (10mmodell) p:05High>03Normal [17:45:42] !log fixed puppet repo rebasing [17:45:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:47:11] RECOVERY - Free space - all mounts on deployment-tin is OK: OK: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found) [18:14:34] PROBLEM - Puppet errors on deployment-deploy01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:28:54] hello releng [18:29:03] https://wikitech.wikimedia.org/wiki/Deployment.eqiad.wmnet exists [18:29:25] but not https://wikitech.wikimedia.org/wiki/Deployment-tin.deployment-prep.eqiad.wmflabs [18:29:29] what's up with that? [18:30:11] i was trying to verify the ssh host key, but needless to say https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/deployment-tin.deployment-prep.eqiad.wmflabs also doesn't exist [18:30:52] cscott: server is no longer tin, it's deploy1001, tin was decommed recently [18:31:13] but deployment-tin is still used to deploy to beta [18:31:26] (deployment-tin, not tin) [18:31:43] ah, sorry, misread [18:35:26] i created https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/deployment-tin.deployment-prep.eqiad.wmflabs but the page needs to be fully-protected [18:37:44] 18<cscott18> what's up with that? [18:37:47] 10Beta-Cluster-Infrastructure: Document deployment-tin.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T198133#4312994 (10cscott) [18:38:07] deployment.eqiad.wmnet isn't an actual server IIRC [18:38:10] it's just a CNAME [18:38:21] to deploy1001.eqiad.wmnet [18:38:30] Krenair: not tin, deployment-tin [18:38:43] you misread the same way thcipriani did :) [18:38:45] I'm talking about production, there is no deployment-tin [18:39:00] $ ssh -A deployment-tin.deployment-prep.eqiad.wmflabs [18:39:00] Linux deployment-tin 4.9.0-0.bpo.6-amd64 #1 SMP Debian 4.9.88-1+deb9u1~bpo8+1 (2018-05-13) x86_64 [18:39:00] Debian GNU/Linux 8.10 (jessie) [18:39:01] The last Puppet run was at Mon Jun 25 18:13:39 UTC 2018 (11 minutes ago). [18:39:01] Last login: Mon Jun 25 18:00:46 2018 from bastion-01.bastion.eqiad.wmflabs [18:39:04] cscott@deployment-tin:~$ [18:39:09] that's .wmflabs [18:39:10] not .wmnet [18:39:15] the machine certainly thinks it is named deployment-tin [18:39:26] Krenair: yes, that was correct in my question above [18:39:31] also holy **** you're agent forwarding to a labs server? [18:40:20] please tell me you don't have your prod key in there [18:40:25] it used to be required by the old parsoid deploy process, we should probably take that out of https://wikitech.wikimedia.org/wiki/Parsoid#Deploying_changes [18:41:02] no, separate keys for prod and labs and 'IdentitiesOnly yes' [18:41:46] I don't know if IdentitiesOnly secures the agent forwarding [18:44:43] strictly speaking, agent forwarding is only dangerous if wmf infra is already compromised, or if you ssh *from* labs/prod *to* some other untrusted host [18:44:51] https://unix.stackexchange.com/questions/70709/ssh-agent-dont-forward-authentication-for-the-whole-keyring#comment105249_71660 suggests it does not [18:45:32] labs is to be treated as compromised for these purposes cscott [18:46:32] if you have your prod key in your agent and then SSH into labs with agent forwarding enabled, your prod key can be used by others [18:46:37] slightly random aside: a while ago I wrote a python script that creates separate ssh-agents for each key as you use them: https://github.com/thcipriani/sshecret [18:47:54] fair enough. [18:48:02] anyway [18:48:20] i can't easily tell from https://wikitech.wikimedia.org/w/index.php?title=Parsoid&action=history how long the -A has been there. [18:48:25] prod has deployment.eqiad.wmnet as a CNAME to the current deployment server [18:48:37] no equivalent exists for deployment-prep [18:48:59] there are multiple deployment hosts but no CNAME to the current one [18:49:58] there is still a machine, whose name is deployment-tin, who is not documented on wiki AFAICT. [18:50:09] CNAME doesn't seem to have anything to do with it [18:50:55] so what you really mean is https://wikitech.wikimedia.org/wiki/Deploy1001 should have a deployment-prep equivalent [18:51:16] I don't think we've ever tried to maintain prod-like documentation for deployment-prep [18:52:48] the wiki page could just say that ;) [18:53:01] ...and link the the SSH fingerprint for the host [18:53:16] I created https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/deployment-tin.deployment-prep.eqiad.wmflabs but it needs to be protected by a sysop [18:53:19] very few labs hosts get SSH fingerprint pages cscott [18:53:23] normally just the bastions [18:53:34] https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints lists quite a few [18:53:41] oh, you said labs hosts [18:54:21] primary.bastion.wmflabs.org, restricted.bastion.wmflabs.org, tools-dev.wmflabs.org, tools-login.wmflabs.org [18:54:31] these are all bastions [18:54:36] deployment-prep does not contain any [18:55:02] 10Beta-Cluster-Infrastructure: Document deployment-tin.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T198133#4313044 (10cscott) I created the SSH finger print page, but it ought to be fully-protected by a sysop on wikitech. [18:55:56] also why is hooft still listed on that page [18:56:11] didn't that get renamed years ago to bast3001? [18:56:14] and/or replaced? [18:57:27] don't ask me! [18:57:53] incidentally, in my testing the `-a` in the ProxyCommand config overrides the -A on the command-line [18:58:11] so no connection to the agent makes it to the bastion [19:00:07] nevermind, i was reading the `ssh -vv` output wrong, i think [19:01:36] we should be setting `AllowAgentForwarding no` in /etc/ssh/sshd_config on the bastions, though [19:02:58] maybe we already are? ssh -vv output is very hard to read. :( [19:03:36] anyway: https://wikitech.wikimedia.org/w/index.php?title=Parsoid&type=revision&diff=1795346&oldid=1794055 [19:03:39] I think it's okay for people who don't have NDAs etc. [19:11:06] 10Beta-Cluster-Infrastructure: Document deployment-tin.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T198133#4313082 (10greg) 05Open>03Resolved p:05Triage>03Lowest a:03cscott Traditionally Cloud VPS project vms don't have similar documentation to production's hosts. Thanks for makin... [19:14:23] Heads up :) - Not sure whether this test was written in pair with those who maintain preferences, but this Selenium test should be disabled if not fixed soon-ish / https://phabricator.wikimedia.org/T198137 [19:33:59] Hello lovely RelEng people, could one of you review https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/441523 ? It'd be good to get it merged before the train tomorrow is cut, though it's not terminal. [19:40:10] (03PS2) 10Greg Grossmeier: Stop branching the MwEmbedSupport extension [tools/release] - 10https://gerrit.wikimedia.org/r/441523 (https://phabricator.wikimedia.org/T197918) (owner: 10Jforrester) [19:43:15] James_F: I'm confused, the first mw-config patch says "This code has moved into the TimedMediaHandler extension directly [19:43:17] from version 1.32.0-wmf.10 onwards. [19:43:26] " ... but we want to stop branching it on wmf.9? [19:43:37] gah, last week, numbers, yes [19:43:38] greg-g: wmf.9 doesn't exist and never will. [19:43:39] 10 [19:43:44] :) [19:43:57] Yup. Right now it's in master, and will be in wmf.10. [19:44:07] the "every week gets a number" makes sense except for when it confuses you :) [19:44:12] :-D [19:44:29] You think /you're/ confused? ReleaseTaggerBot hasn't run since wmf.999 went live, AFAICT. [19:44:31] ack, makes sense now. [19:44:45] poor releasetaggerbot :( [19:45:28] It'll get magically fixed as soon as group 0 rolls over to wmf.10. I think. [19:45:34] so, we're sure we got all the calls to this extension migrated? [19:46:08] Yes. I ran it fine on my dev machine with and without any MES repo and it worked exactly the same. [19:46:22] I would say "well", but that would suggest that TMH is good code, which isn't… true. [19:46:29] :( [19:46:42] But with this change, better. [19:48:02] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:48:38] confused about this https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TimedMediaHandler/+/441522/ breakage :( [19:49:27] greg-g it happens to other ext too [19:49:39] i was speaking to hashar about translate [19:49:45] which dosen't get it's deps either [19:50:06] even though they are specified in integration/config [19:51:24] PROBLEM - Host deployment-mx is DOWN: CRITICAL - Host Unreachable (10.68.17.78) [19:54:24] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:54:56] (03CR) 10Greg Grossmeier: [C: 032] Stop branching the MwEmbedSupport extension [tools/release] - 10https://gerrit.wikimedia.org/r/441523 (https://phabricator.wikimedia.org/T197918) (owner: 10Jforrester) [19:55:27] ack, so if it's happening in other places we should fix the underlying issue and not let that block this change. +2'd [19:55:48] (03Merged) 10jenkins-bot: Stop branching the MwEmbedSupport extension [tools/release] - 10https://gerrit.wikimedia.org/r/441523 (https://phabricator.wikimedia.org/T197918) (owner: 10Jforrester) [20:16:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4313307 (10hashar) [20:22:35] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MediaWiki-User-preferences, 10Wikimedia-log-errors (Shared Build Failure): Selenium "User should be able to change preferences" test flaky - https://phabricator.wikimedia.org/T198137#4313316 (10greg) a:03zeljkofilipin Looks like it was mostly... [20:23:02] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [20:43:46] paladox: that TimedMediaHandler handler issue is most probably the same as the Translate one you mentioned earlier [20:43:54] yep [20:52:00] 10Release-Engineering-Team, 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Selenium test job should install local dependencies before starting tests - https://phabricator.wikimedia.org/T193943#4313350 (10Krinkle) @zeljkofilipin Yeah, misunderstanding. But I realise I did forget to address the `specs` discover... [20:53:05] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Document differences between Ruby and Node.js Selenium frameworks - https://phabricator.wikimedia.org/T182692#4313353 (10Krinkle) @zeljkofilipin @hashar I replied at T193943, which seems to be about the same. See T193943#4313350. [20:53:47] hashar: sorry for the long text, was not able to make it shorter. I hope it makes sense :) - My goal is for you and Zjelko to have less work with selenium! [20:54:28] Željko * [20:54:46] Krinkle: I noticed the wdio-mediawiki node module already :] [20:55:59] Krinkle: I will catch up tomorrow hopefully, but the reasonning was to have everything in core at start, much like the qunit tests are run from core [20:57:05] zeljko and I talked this mornign about running them standalone, one such change got merged today iirc and yeah eventually we will have a docker container that clones the extension and run npm install && npm run-script selenium-something [21:12:20] (03PS1) 10Hashar: Skip selenium tests on some repositories [integration/config] - 10https://gerrit.wikimedia.org/r/441984 (https://phabricator.wikimedia.org/T196960) [21:14:41] (03CR) 10Hashar: "I would love a better way to skip selenium tests, but could not find anything better for now. The reason I have choose to send --skip=se" [integration/config] - 10https://gerrit.wikimedia.org/r/441984 (https://phabricator.wikimedia.org/T196960) (owner: 10Hashar) [21:15:11] 10Continuous-Integration-Infrastructure (shipyard), 10MediaWiki-Core-Tests, 10Quibble, 10Patch-For-Review, 10User-zeljkofilipin: Quibble should have a way for extensions to opt out of core selenium browser tests - https://phabricator.wikimedia.org/T196960#4313388 (10hashar) [21:15:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Quibble, and 2 others: Quibble should have a way for extensions to opt out of core selenium browser tests - https://phabricator.wikimedia.org/T196960#4313390 (10hashar) a:03hashar [21:19:47] !log deployed to beta: [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52 [21:19:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:32:40] hi [21:32:42] b631ec6ae5..db8bc87174 wmf/1.32.0-wmf.999 -> origin/wmf/1.32.0-wmf.999 [21:32:44] 999 ? [21:55:17] (03CR) 10Legoktm: [C: 04-1] "This is a soft -1, because I'd like to see this controlled in the extension repository itself instead of the CI configuration." [integration/config] - 10https://gerrit.wikimedia.org/r/441984 (https://phabricator.wikimedia.org/T196960) (owner: 10Hashar) [21:56:04] legoktm: You're back? Or just working too hard? :-) [21:56:47] (03CR) 10Legoktm: [C: 04-1] "We could have a .quibble.json file that allows for skipping stages or something?" [integration/config] - 10https://gerrit.wikimedia.org/r/441984 (https://phabricator.wikimedia.org/T196960) (owner: 10Hashar) [22:55:18] (03PS1) 10MarcoAurelio: Edit Project Config [extensions/CentralAuth] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/441995 [22:55:52] (03Abandoned) 10MarcoAurelio: Edit Project Config [extensions/CentralAuth] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/441995 (owner: 10MarcoAurelio) [23:42:01] (03PS1) 10Arlolra: Test Parsoid's version of Poem extension's parserTests against MediaWiki [integration/config] - 10https://gerrit.wikimedia.org/r/442002 [23:42:32] (03PS2) 10Arlolra: Test Parsoid's version of Poem extension's parserTests against MediaWiki [integration/config] - 10https://gerrit.wikimedia.org/r/442002 [23:48:48] (03PS1) 10Arlolra: Run Parsoid's langParserTests against MediaWiki [integration/config] - 10https://gerrit.wikimedia.org/r/442004 [23:53:02] (03CR) 10Subramanya Sastry: [C: 031] Run Parsoid's langParserTests against MediaWiki [integration/config] - 10https://gerrit.wikimedia.org/r/442004 (owner: 10Arlolra)