[00:16:40] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10serviceops: Add second virtual hard disk to ganeti gerrit test instance - https://phabricator.wikimedia.org/T243983 (10Dzahn) VM is back up now and has an additional 10GB drive (/dev/vd... [00:16:45] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10serviceops: Add second virtual hard disk to ganeti gerrit test instance - https://phabricator.wikimedia.org/T243983 (10Dzahn) 05Open→03Resolved [00:17:50] James_F: fixed it, wrong interface in /etc/network/interfaces. new disk is mounted on /srv/dbdump and added in /etc/fstab ..so everything should survice reboots. the DB should not be larger than 2GB [00:20:55] Nice. [00:20:58] Thank you! [00:31:52] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Core-Testing, 10Performance-Team, and 3 others: ERROR webdriver: Request failed due to Error: session not created: Chrome version must b... - https://phabricator.wikimedia.org/T234610 [01:29:41] 10Beta-Cluster-Infrastructure, 10ContentSecurityPolicy, 10Maps, 10Patch-For-Review: Beta Cluster points to maps-beta.wmflabs.org instead of maps.wikimedia.org, which CSP blocks - https://phabricator.wikimedia.org/T240960 (10Bawolff) It is kind of useful to not have equiv domains. It makes it easier to find... [01:36:45] (03PS2) 10Jforrester: layout: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker fully [integration/config] - 10https://gerrit.wikimedia.org/r/569635 (https://phabricator.wikimedia.org/T243975) [01:37:39] (03CR) 10jerkins-bot: [V: 04-1] layout: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker fully [integration/config] - 10https://gerrit.wikimedia.org/r/569635 (https://phabricator.wikimedia.org/T243975) (owner: 10Jforrester) [01:38:47] (03PS3) 10Jforrester: layout: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker fully [integration/config] - 10https://gerrit.wikimedia.org/r/569635 (https://phabricator.wikimedia.org/T243975) [01:40:59] (03CR) 10Jforrester: [C: 03+2] layout: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker fully [integration/config] - 10https://gerrit.wikimedia.org/r/569635 (https://phabricator.wikimedia.org/T243975) (owner: 10Jforrester) [01:41:55] (03Merged) 10jenkins-bot: layout: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker fully [integration/config] - 10https://gerrit.wikimedia.org/r/569635 (https://phabricator.wikimedia.org/T243975) (owner: 10Jforrester) [01:42:33] !log Zuul: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker always T243975 [01:42:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:42:35] T243975: Run API integration tests suite in CI on core repo without extensions - https://phabricator.wikimedia.org/T243975 [01:42:51] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Core Platform Team, 10CPT Initiatives (API Integration Tests), and 2 others: Run API integration tests suite in CI on core repo without extensions - https://phabricator.wikimedia.org/T243975 (10Jdforrester-WMF) 0... [01:42:53] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10CPT Initiatives (API Integration Tests), 10Code-Health, and 2 others: Enable API integration tests in CI for MediaWiki core - https://phabricator.wikimedia.org/T236680 (10Jdforrester-WMF) [01:54:42] (03PS1) 10Jforrester: layout: [wikimedia/portals] Add service-pipeline configuration [integration/config] - 10https://gerrit.wikimedia.org/r/569694 (https://phabricator.wikimedia.org/T238747) [01:55:01] (03CR) 10Jforrester: [C: 03+2] "Let's see what happens." [integration/config] - 10https://gerrit.wikimedia.org/r/569694 (https://phabricator.wikimedia.org/T238747) (owner: 10Jforrester) [01:56:09] (03Merged) 10jenkins-bot: layout: [wikimedia/portals] Add service-pipeline configuration [integration/config] - 10https://gerrit.wikimedia.org/r/569694 (https://phabricator.wikimedia.org/T238747) (owner: 10Jforrester) [01:56:31] !log Zuul: [wikimedia/portals] Add service-pipeline configuration T238747 [01:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:56:33] T238747: Migrate www.wikimedia.org (the portal) to be hosted as a service - https://phabricator.wikimedia.org/T238747 [02:24:46] (03PS1) 10Gopavasanth: Merge branch 'master' of https://gerrit.wikimedia.org/r/integration/config [integration/config] - 10https://gerrit.wikimedia.org/r/569696 [02:24:47] (03PS1) 10Gopavasanth: Add labs/tools/video-cut-tool-back-end [integration/config] - 10https://gerrit.wikimedia.org/r/569697 (https://phabricator.wikimedia.org/T244079) [03:54:53] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T233864 (10brennen) [04:12:24] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Tgr) >>! In T234743#5843756, @Zache wrote: > And also as catrope [[https://phabricator.wikimedia.org/T234743#56... [04:16:09] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T233864 (10brennen) [04:19:44] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen: 1.35.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T233864 (10brennen) 05Open→03Resolved wmf.16 deployed to all wikis at 2020-02-03 23:24 UTC. [04:32:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Upgrade integration/config to use Fabric 2.x / python3 - https://phabricator.wikimedia.org/T236689 (10Legoktm) >>! In T236689#5611791, @hashar wrote: > I went to Fabric website ( http... [04:38:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Python3-Porting: Upgrade integration/config to use Fabric 2.x / python3 - https://phabricator.wikimedia.org/T236689 (10Legoktm) a:03Legoktm >>! In T236689#5637835, @hashar wrote... [05:06:42] I forgot how awful fabric2 is [05:19:57] (03PS1) 10Legoktm: [WIP] Port fabfile.py to fabric 2 / Python 3 [integration/config] - 10https://gerrit.wikimedia.org/r/569703 (https://phabricator.wikimedia.org/T236689) [05:27:21] nothing like replacing a 10k+ dependency with 3 lines of code [05:27:30] 10k+ lines of code dependency* [05:28:16] (03PS2) 10Legoktm: [WIP] Port fabfile.py to fabric 2 / Python 3 [integration/config] - 10https://gerrit.wikimedia.org/r/569703 (https://phabricator.wikimedia.org/T236689) [05:57:43] (03PS2) 10Gopavasanth: Add labs/tools/video-cut-tool-back-end [integration/config] - 10https://gerrit.wikimedia.org/r/569696 (https://phabricator.wikimedia.org/T244079) [06:03:16] (03PS3) 10Gopavasanth: Add labs/tools/video-cut-tool-back-end [integration/config] - 10https://gerrit.wikimedia.org/r/569696 (https://phabricator.wikimedia.org/T244079) [06:11:58] (03CR) 10Gopavasanth: [C: 04-1] "> Main test build succeeded." [integration/config] - 10https://gerrit.wikimedia.org/r/569696 (https://phabricator.wikimedia.org/T244079) (owner: 10Gopavasanth) [06:13:25] (03Abandoned) 10Gopavasanth: Add labs/tools/video-cut-tool-back-end [integration/config] - 10https://gerrit.wikimedia.org/r/569696 (https://phabricator.wikimedia.org/T244079) (owner: 10Gopavasanth) [09:09:32] (03PS1) 10Legoktm: Expand README [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569837 [09:09:34] (03PS1) 10Legoktm: Install requirements.txt for pytest if it exists [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569838 [09:09:36] (03PS1) 10Legoktm: Add mypy support [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569839 [09:09:38] (03PS1) 10Legoktm: Officially support Python 3.8 [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 [09:09:40] (03PS1) 10Legoktm: Run mypy against this repository [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569841 [09:13:45] (03CR) 10jerkins-bot: [V: 04-1] Add mypy support [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569839 (owner: 10Legoktm) [09:14:01] (03CR) 10jerkins-bot: [V: 04-1] Officially support Python 3.8 [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 (owner: 10Legoktm) [09:14:14] (03CR) 10jerkins-bot: [V: 04-1] Run mypy against this repository [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569841 (owner: 10Legoktm) [09:14:22] arg [09:15:13] (03PS2) 10Legoktm: Add mypy support [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569839 [09:15:15] (03PS2) 10Legoktm: Officially support Python 3.8 [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 [09:15:17] (03PS2) 10Legoktm: Run mypy against this repository [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569841 [09:15:30] (03CR) 10jerkins-bot: [V: 04-1] Officially support Python 3.8 [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 (owner: 10Legoktm) [09:16:00] (03CR) 10Legoktm: "recheck" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 (owner: 10Legoktm) [09:22:35] 10Beta-Cluster-Infrastructure, 10ContentSecurityPolicy, 10Maps, 10Patch-For-Review: Beta Cluster points to maps-beta.wmflabs.org instead of maps.wikimedia.org, which CSP blocks - https://phabricator.wikimedia.org/T240960 (10Krenair) I'm not sure I follow. Why would it need prod domains for that? Wouldn't i... [09:29:04] (03PS1) 10Legoktm: Fix variable name collision with "config" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569845 [09:29:06] (03PS1) 10Legoktm: Add descriptions of tools for `tox -av` [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569846 [09:31:33] (03CR) 10Jdlrobson: "3 years later same question as 2016. Still waiting?" [integration/docroot] - 10https://gerrit.wikimedia.org/r/131324 (owner: 10Jforrester) [09:32:04] (03CR) 10Jdlrobson: "friendly ping that this is still open" [integration/docroot] - 10https://gerrit.wikimedia.org/r/131323 (owner: 10Jforrester) [09:33:49] (03CR) 10jerkins-bot: [V: 04-1] Fix variable name collision with "config" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569845 (owner: 10Legoktm) [09:34:24] (03CR) 10jerkins-bot: [V: 04-1] Add descriptions of tools for `tox -av` [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569846 (owner: 10Legoktm) [10:18:01] 10Beta-Cluster-Infrastructure, 10ContentSecurityPolicy, 10Maps, 10Patch-For-Review: Beta Cluster points to maps-beta.wmflabs.org instead of maps.wikimedia.org, which CSP blocks - https://phabricator.wikimedia.org/T240960 (10Bawolff) Sorry. I mean to say- i dont think we should replace the prod csp domains... [12:59:05] PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash2.diskspace._mnt.byte_percentfree (No valid datapoints found) deployment-prep.deployment-logstash2.diskspace._var_lib_elasticsearch.byte_percentfree (No valid datapoints found)deployment-prep.deployment-logstash2.diskspace.root.byte_percentfree (<100.00%) [14:17:29] (03CR) 10Gopavasanth: "> Change has been successfully merged by jenkins-bot" [integration/config] - 10https://gerrit.wikimedia.org/r/569352 (https://phabricator.wikimedia.org/T244079) (owner: 10Gopavasanth) [14:20:16] 10Continuous-Integration-Config, 10VideoCutTool, 10Patch-For-Review: Set up continuous integration for VideoCutTool - https://phabricator.wikimedia.org/T244079 (10Gopavasanth) After merging this [[ https://gerrit.wikimedia.org/r/#/c/integration/config/+/569352/ | patch ]] Jenkins is always failing in every... [14:47:09] 10Continuous-Integration-Config, 10VideoCutTool, 10Patch-For-Review: Set up continuous integration for VideoCutTool - https://phabricator.wikimedia.org/T244079 (10Urbanecm) Hi there, I don't see any tox.ini file in your repo. You have configured CI to run tox, but not told tox what it should actually test. I... [16:04:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:32] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-parsoid10 is CRITICAL: (Service Check Timed Out) [16:05:48] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:30] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:07:01] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-07 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:11] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 91542 bytes in 1.617 second response time [16:14:24] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-parsoid10 is OK: HTTP OK: HTTP/1.1 200 OK - 91570 bytes in 1.308 second response time [16:15:39] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 51402 bytes in 3.206 second response time [16:16:23] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 92000 bytes in 3.070 second response time [16:16:56] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-07 is OK: HTTP OK: HTTP/1.1 200 OK - 91542 bytes in 3.256 second response time [16:30:20] PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:12] RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 91548 bytes in 1.099 second response time [16:57:07] 10Phabricator, 10Security-Team, 10Security: Convert #Security to acl*Security_Task_Access - https://phabricator.wikimedia.org/T244165 (10chasemp) >>! In T244165#5845967, @sbassett wrote: >>>! In T244165#5845944, @chasemp wrote: >>> volunteer >>> - Tchanders > > I believe they are wmf, see: https://www.media... [17:01:36] 10Phabricator, 10Security-Team, 10Privacy: Turn off #privacy workboard - https://phabricator.wikimedia.org/T244161 (10JFishback_WMF) @Krinkle if I understand this correctly, resurrecting the `privacy` tag as an alias for `privacy_engineering` is not what we are ultimately looking for. We want `privacy_engine... [17:04:20] 10Phabricator, 10Security-Team, 10Privacy: Turn off #privacy workboard - https://phabricator.wikimedia.org/T244161 (10Krinkle) I'm merely proposing that if someone follows an old url like or if they have some other process in place that instead of ending up so... [17:04:22] 10Phabricator, 10Security-Team, 10Privacy: Turn off #privacy workboard - https://phabricator.wikimedia.org/T244161 (10chasemp) >>! In T244161#5848764, @JFishback_WMF wrote: > @Krinkle if I understand this correctly, resurrecting the `privacy` tag as an alias for `privacy_engineering` is not what we are ultim... [17:13:49] 10Phabricator, 10Security-Team, 10Privacy: Turn off #privacy workboard - https://phabricator.wikimedia.org/T244161 (10Aklapper) @Krinkle: Sounds like there is a misunderstanding here? #privacy remains #privacy. This task is only about disabling the workboard of #privacy. This is not about disabling the #priv... [17:16:03] 10Phabricator, 10Security-Team, 10Privacy: Turn off #privacy workboard - https://phabricator.wikimedia.org/T244161 (10Krinkle) I indeed thought the plan was to archive the tag. Nevermind. Thanks for clarifying, @Aklapper! [17:31:36] (03CR) 10Jforrester: "> Patch Set 4:" [integration/config] - 10https://gerrit.wikimedia.org/r/569352 (https://phabricator.wikimedia.org/T244079) (owner: 10Gopavasanth) [18:08:16] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Prep short pres on new ci threat modelling - https://phabricator.wikimedia.org/T242963 (10LarsWirzenius) 05Open→03Resolved [18:08:36] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Delete Jenkins label DebianJessie - https://phabricator.wikimedia.org/T239981 (10Jdforrester-WMF) [18:08:39] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [18:13:40] (03PS1) 10Krinkle: Revert "actually commit doc.wm.o favicon" [integration/docroot] - 10https://gerrit.wikimedia.org/r/570105 [18:14:06] (03PS2) 10Krinkle: Revert "actually commit doc.wm.o favicon" [integration/docroot] - 10https://gerrit.wikimedia.org/r/570105 [18:14:21] (03PS3) 10Krinkle: docs: Minor README change [integration/docroot] - 10https://gerrit.wikimedia.org/r/570105 [18:14:29] (03PS4) 10Krinkle: docs: Minor README change [integration/docroot] - 10https://gerrit.wikimedia.org/r/570105 [18:14:33] (03CR) 10Krinkle: [C: 03+2] docs: Minor README change [integration/docroot] - 10https://gerrit.wikimedia.org/r/570105 (owner: 10Krinkle) [18:14:45] (03CR) 10Krinkle: [C: 03+2] "Testing permissions for T237707." [integration/docroot] - 10https://gerrit.wikimedia.org/r/570105 (owner: 10Krinkle) [18:15:17] (03Merged) 10jenkins-bot: docs: Minor README change [integration/docroot] - 10https://gerrit.wikimedia.org/r/570105 (owner: 10Krinkle) [18:24:01] (03CR) 10Jforrester: "Trying this out throws "sudo: cd: command not found"…" [integration/config] - 10https://gerrit.wikimedia.org/r/569703 (https://phabricator.wikimedia.org/T236689) (owner: 10Legoktm) [18:59:43] (03PS2) 10Umherirrender: Inline mostly meaningless $error/$data variables in many sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/567438 (owner: 10Thiemo Kreuz (WMDE)) [18:59:47] (03CR) 10Umherirrender: [C: 03+2] Inline mostly meaningless $error/$data variables in many sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/567438 (owner: 10Thiemo Kreuz (WMDE)) [19:00:24] (03Merged) 10jenkins-bot: Inline mostly meaningless $error/$data variables in many sniffs [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/567438 (owner: 10Thiemo Kreuz (WMDE)) [19:09:08] (03PS3) 10Jforrester: jjb: Create wikidata-query-rdf-release-silent [integration/config] - 10https://gerrit.wikimedia.org/r/567068 (https://phabricator.wikimedia.org/T243603) (owner: 10ZPapierski) [19:09:14] (03PS4) 10Jforrester: jjb: Create wikidata-query-rdf-release-silent [integration/config] - 10https://gerrit.wikimedia.org/r/567068 (https://phabricator.wikimedia.org/T243603) (owner: 10ZPapierski) [19:09:20] (03CR) 10Jforrester: [C: 03+2] jjb: Create wikidata-query-rdf-release-silent [integration/config] - 10https://gerrit.wikimedia.org/r/567068 (https://phabricator.wikimedia.org/T243603) (owner: 10ZPapierski) [19:10:17] (03Merged) 10jenkins-bot: jjb: Create wikidata-query-rdf-release-silent [integration/config] - 10https://gerrit.wikimedia.org/r/567068 (https://phabricator.wikimedia.org/T243603) (owner: 10ZPapierski) [19:10:24] PROBLEM - Free space - all mounts on deployment-logstash03 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash03.diskspace.root.byte_percentfree (<100.00%) [19:11:15] (03CR) 10Jforrester: "I've deployed this for you so you can make progress, but you're going to have to migrate it to a real docker image very soon (at most, wee" [integration/config] - 10https://gerrit.wikimedia.org/r/567068 (https://phabricator.wikimedia.org/T243603) (owner: 10ZPapierski) [19:50:15] (03PS2) 10Umherirrender: Remove little bits of unused code from ForbiddenFunctions sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/567440 (owner: 10Thiemo Kreuz (WMDE)) [19:50:18] (03CR) 10Umherirrender: [C: 03+2] Remove little bits of unused code from ForbiddenFunctions sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/567440 (owner: 10Thiemo Kreuz (WMDE)) [19:50:38] (03Merged) 10jenkins-bot: Remove little bits of unused code from ForbiddenFunctions sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/567440 (owner: 10Thiemo Kreuz (WMDE)) [20:47:55] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T233866 (10mmodell) branch cut for wmf.18: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/570138 [21:22:46] 10Beta-Cluster-Infrastructure, 10ContentSecurityPolicy, 10Maps, 10Patch-For-Review: Beta Cluster points to maps-beta.wmflabs.org instead of maps.wikimedia.org, which CSP blocks - https://phabricator.wikimedia.org/T240960 (10Krenair) >>! In T240960#5847501, @Bawolff wrote: > not having beta ones eases detec... [21:49:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10mmodell) What's the current status of this? Is this still a train blocker? [21:57:31] 10Beta-Cluster-Infrastructure, 10ContentSecurityPolicy, 10Maps, 10Patch-For-Review: Beta Cluster points to maps-beta.wmflabs.org instead of maps.wikimedia.org, which CSP blocks - https://phabricator.wikimedia.org/T240960 (10Krinkle) @Bawolff MediaWiki extensions that interact with Maps have their values in... [22:55:33] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T233866 (10Reedy) [22:56:06] 10Release-Engineering-Team, 10Performance-Team, 10serviceops: Create warmup procedure for MediaWiki app servers - https://phabricator.wikimedia.org/T230037 (10Krinkle) During the first three switch overs the impact and gains was imho quite clearly proven given that on a cold server (at the time HHVM), latenc... [23:06:45] (03PS2) 10Legoktm: Fix variable name collision with "config" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569845 [23:06:47] (03PS2) 10Legoktm: Add descriptions of tools for `tox -av` [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569846 [23:07:46] (03CR) 10Legoktm: [C: 03+2] Expand README [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569837 (owner: 10Legoktm) [23:08:12] (03CR) 10Legoktm: [C: 03+2] Officially support Python 3.8 [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 (owner: 10Legoktm) [23:11:04] (03Merged) 10jenkins-bot: Expand README [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569837 (owner: 10Legoktm) [23:21:56] (03CR) 10Volans: [C: 03+1] "LGTM, one question/comment inline" (031 comment) [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569838 (owner: 10Legoktm) [23:28:32] !log Puppet agent has not been able to run on deployment-mediawiki-07 for 2.5 days since 2 Feb 18:27 [23:28:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:29:43] 10Beta-Cluster-Infrastructure, 10Operations: Puppet broken on Beta Cluster app server - https://phabricator.wikimedia.org/T244306 (10Krinkle) [23:30:35] Krinkle: possibly related to T244306 and possibly T243226 [23:30:36] T243226: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 [23:30:36] T244306: Puppet broken on Beta Cluster app server - https://phabricator.wikimedia.org/T244306 [23:30:56] 10Beta-Cluster-Infrastructure, 10Operations: Puppet broken on Beta Cluster app server - https://phabricator.wikimedia.org/T244306 (10Krinkle) [23:31:32] 10Beta-Cluster-Infrastructure, 10Operations: Puppet broken on Beta Cluster app server - https://phabricator.wikimedia.org/T244306 (10Krenair) [23:31:34] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep - https://phabricator.wikimedia.org/T243226 (10Krenair) [23:32:44] Reedy: that first one is the one I just filed after !log ;-) [23:32:45] Thanks [23:33:14] oh hi [23:33:16] yeah I know about it [23:33:28] Is somene working on it with you / do you need anything? [23:33:36] waiting for someone to port puppetdb-termini to have the correct version available on stretch [23:33:38] need someone in ops [23:33:54] how did this not break prod? [23:33:57] which puppet.git commit caused it? [23:34:08] prod has a buster puppetmaster [23:34:15] none [23:34:23] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10Krinkle) [23:34:35] Krenair: I mean, thigns were okay-ish until 2 days ago What changed? [23:34:40] T244074 was the other task [23:34:41] T244074: Host key verification failed x 2 - https://phabricator.wikimedia.org/T244074 [23:35:11] Krinkle, I followed https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Upgrading_to_puppet_5.5 [23:35:32] Reedy, yes, did that do the trick? [23:35:49] Krenair: Seemingly that unbroke the CI deployment jobs, yeah [23:35:58] As the bot isn't complaining still [23:36:03] I see. So the change is a new puppetmaster not yet being in working state but being enabled as the canonical one. [23:36:03] 10Beta-Cluster-Infrastructure: Host key verification failed x 2 - https://phabricator.wikimedia.org/T244074 (10Krenair) 05Open→03Resolved Krenair: Seemingly that unbroke the CI deployment jobs, yeah [23:36:11] Is the old one still running? [23:36:16] no [23:36:22] there is no old one [23:36:25] it's the same puppetmaster [23:36:29] So what changed? [23:36:33] as I said I followed https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Upgrading_to_puppet_5.5 [23:36:49] did prod upgrade 2 days ago? [23:36:50] it's the same puppetmaster as we've had for months [23:36:51] no [23:37:18] did we need to upgrade beta 2 days ago? Im just trying to figure out reasons, I'm not disagreeing :) [23:37:49] Krenair: do you mean you upgraded beta's puppet to 5.5 2 days ago? [23:38:09] puppet had been failing on a bunch of servers, people think due to some puppet version incompatibility with prod's latest manifests. so I went to do the upgrade [23:38:21] yes [23:38:31] I see. So prod had been on 5.5 longer, but so far didn't break beta's app servers. [23:38:36] but did break other services [23:38:42] and now it's the other way around [23:38:43] okay [23:38:45] I mean [23:39:12] at least one beta's app servers had been not running puppet for other reasons until I started working on the upgrade task [23:39:25] for a lot longer than 2 days [23:39:32] thats actually what led me to check on it now because I had a todo item to look into that. [23:39:43] Do you recall what that was? [23:40:10] My last recorded failure was https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/561816/ [23:40:18] which was followed-up but I'm not sure it actually worked [23:40:32] yes, someone had disabled puppet on it and left it for approx 12 days [23:40:41] ok [23:40:49] I'll move on then and pretend everything's fine. [23:41:03] Thanks for working on this Krenair, I mean it :) [23:42:30] right now puppet is entirely broken across the whole project [23:42:51] we could either make a buster puppetmaster and re-certify everything (~80 hosts IIRC) [23:43:08] or wait for ops to port a missing package [23:43:37] I'm doing the latter [23:43:52] I suspect the cause of the issue is that the puppet upgrade process was tested without puppetdb [23:45:53] thx, I'll try to get eyes on it [23:45:58] I'll get around to the former option eventually [23:46:18] I just have a bunch of other things to be doing [23:46:30] (03CR) 10Volans: [C: 03+1] "LGTM, a nit and a philosophical question inline" (032 comments) [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569839 (owner: 10Legoktm) [23:46:48] (03CR) 10Volans: [C: 03+1] "LGTM" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 (owner: 10Legoktm) [23:47:52] (03CR) 10Volans: [C: 03+1] "LGTM" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569841 (owner: 10Legoktm) [23:48:26] (03CR) 10Volans: [C: 03+1] "LGTM" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569845 (owner: 10Legoktm) [23:48:53] (03CR) 10Volans: [C: 03+1] "LGTM" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569846 (owner: 10Legoktm) [23:50:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10bd808) Happy bug birthday {T215217}! [23:53:07] 10Release-Engineering-Team-TODO, 10MediaWiki-Core-Testing, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review: Upgrade MW core from karma 3.x to 4.x - https://phabricator.wikimedia.org/T222293 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF