[00:00:51] 10Release-Engineering-Team (Watching / External), 10Operations, 10monitoring, 10Performance-Team (Radar), 10goodfirstbug: Increase "check_legal_html" coverage to group0 wikis - https://phabricator.wikimedia.org/T208284 (10greg) [00:00:51] 10Release-Engineering-Team (Watching / External), 10Operations, 10monitoring, 10Performance-Team (Radar), 10goodfirstbug: Increase "check_legal_html" coverage to group0 wikis - https://phabricator.wikimedia.org/T208284 (10greg) [00:01:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10greg) [00:01:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10greg) [00:01:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins: Make the new Jenkins login page match the fancy gerrit login page - https://phabricator.wikimedia.org/T207762 (10greg) [00:01:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins: Make the new Jenkins login page match the fancy gerrit login page - https://phabricator.wikimedia.org/T207762 (10greg) [00:01:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10greg) [00:01:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations: contint1001 store docker images on separate partition or disk - https://phabricator.wikimedia.org/T207707 (10greg) [00:01:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Release Pipeline: contint1001:/var/lib/docker growth - https://phabricator.wikimedia.org/T207702 (10greg) [00:01:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Release Pipeline: contint1001:/var/lib/docker growth - https://phabricator.wikimedia.org/T207702 (10greg) [00:02:05] twentyafterfour: so for T210030 is the hope to reconfigure beta to store sessions somewhere other than redis? Or should we just get the redis instances fixed? [00:02:05] twentyafterfour: so for T210030 is the hope to reconfigure beta to store sessions somewhere other than redis? Or should we just get the redis instances fixed? [00:02:06] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [00:02:06] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [00:02:23] both are possible (I think) [00:02:23] both are possible (I think) [00:02:59] bd808: either way would be fine with me [00:02:59] bd808: either way would be fine with me [00:03:17] several people decided to shut down redis and if it's not necessary then we can save 2 instances [00:03:17] several people decided to shut down redis and if it's not necessary then we can save 2 instances [00:03:38] 10Release-Engineering-Team (Backlog), 10Scap, 10ContentTranslation: cx-entrypoint-dialog-page-doesnt-exist-yet showing
- https://phabricator.wikimedia.org/T207522 (10greg) [00:03:38] 10Release-Engineering-Team (Backlog), 10Scap, 10ContentTranslation: cx-entrypoint-dialog-page-doesnt-exist-yet showing
- https://phabricator.wikimedia.org/T207522 (10greg) [00:04:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Backlog), 10User-Jdlrobson: Automatically display production content on Beta Cluster wikis (via an API request) - https://phabricator.wikimedia.org/T207508 (10greg) [00:04:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Backlog), 10User-Jdlrobson: Automatically display production content on Beta Cluster wikis (via an API request) - https://phabricator.wikimedia.org/T207508 (10greg) [00:05:55] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.32-release: REL1_32 branch didn't get cut for some extensions - https://phabricator.wikimedia.org/T207281 (10greg) a:03mmodell Are we good here? [00:05:55] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.32-release: REL1_32 branch didn't get cut for some extensions - https://phabricator.wikimedia.org/T207281 (10greg) a:03mmodell Are we good here? [00:06:31] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Revisit the logging work done on Q1 2017-2018 for the standard pod setup - https://phabricator.wikimedia.org/T207200 (10greg) [00:06:31] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Revisit the logging work done on Q1 2017-2018 for the standard pod setup - https://phabricator.wikimedia.org/T207200 (10greg) [00:06:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Set up a Prometheus server under the integration project - https://phabricator.wikimedia.org/T205930 (10greg) [00:06:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Set up a Prometheus server under the integration project - https://phabricator.wikimedia.org/T205930 (10greg) [00:06:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Collect and expose Jenkins build metrics for visualization, reporting, and analysis - https://phabricator.wikimedia.org/T205927 (10greg) [00:06:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog): Collect and expose Jenkins build metrics for visualization, reporting, and analysis - https://phabricator.wikimedia.org/T205927 (10greg) [00:07:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): TEC3:O1:O1.2 Goal – Formalize the collection of CI infrastructure and tooling metrics - https://phabricator.wikimedia.org/T205923 (10greg) a:03dduvall [00:07:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): TEC3:O1:O1.2 Goal – Formalize the collection of CI infrastructure and tooling metrics - https://phabricator.wikimedia.org/T205923 (10greg) a:03dduvall [00:08:34] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10greg) [00:08:34] 10Release-Engineering-Team (Watching / External), 10Operations, 10Release Pipeline, 10Core Platform Team Backlog (Watching / External), 10Services (watching): Track and install additional npm packages for all service container images - https://phabricator.wikimedia.org/T205911 (10greg) [00:09:13] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: l10nupdate is still using HHVM - https://phabricator.wikimedia.org/T205313 (10greg) p:05Triage>03Normal a:03thcipriani [00:09:13] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: l10nupdate is still using HHVM - https://phabricator.wikimedia.org/T205313 (10greg) p:05Triage>03Normal a:03thcipriani [00:10:41] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Signing MW Core Tags - https://phabricator.wikimedia.org/T204995 (10greg) a:03mmodell Assigning to @mmodell so he doesn't forget for this release. [00:10:41] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Signing MW Core Tags - https://phabricator.wikimedia.org/T204995 (10greg) a:03mmodell Assigning to @mmodell so he doesn't forget for this release. [00:12:45] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Zuul cancel all changes when a change is manually merged - https://phabricator.wikimedia.org/T203846 (10greg) a:03hashar [00:12:45] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Zuul cancel all changes when a change is manually merged - https://phabricator.wikimedia.org/T203846 (10greg) a:03hashar [00:14:29] twentyafterfour: in wmf-config/InitialiseSettings.php, wgSessionCacheType is set to 'redis_local' for everything other than wikitech. This would need to change to get beta off of redis as a session backend [00:14:29] twentyafterfour: in wmf-config/InitialiseSettings.php, wgSessionCacheType is set to 'redis_local' for everything other than wikitech. This would need to change to get beta off of redis as a session backend [00:14:33] hey [00:14:33] hey [00:14:35] 10Deployments: Consider having an IRC bot to manage the deployment "conch" for out-of-band (and scheduled?) deploys - https://phabricator.wikimedia.org/T203267 (10greg) p:05Triage>03Low [00:14:35] 10Deployments: Consider having an IRC bot to manage the deployment "conch" for out-of-band (and scheduled?) deploys - https://phabricator.wikimedia.org/T203267 (10greg) p:05Triage>03Low [00:14:38] So I just checked [00:14:38] So I just checked [00:14:47] We still have redis_lock entries pointing at redis* hosts [00:14:47] We still have redis_lock entries pointing at redis* hosts [00:14:54] but prod has those pointing at memc hosts [00:14:54] but prod has those pointing at memc hosts [00:15:14] 10Deployments, 10Release-Engineering-Team (Backlog): Add branch committers to train task - https://phabricator.wikimedia.org/T203111 (10greg) p:05Triage>03Normal [00:15:14] 10Deployments, 10Release-Engineering-Team (Backlog): Add branch committers to train task - https://phabricator.wikimedia.org/T203111 (10greg) p:05Triage>03Normal [00:15:15] I'm about to test changing our list to be memc hosts [00:15:15] I'm about to test changing our list to be memc hosts [00:15:36] 10Release-Engineering-Team, 10Scap, 10Epic, 10Goal: Automate the Train - https://phabricator.wikimedia.org/T196515 (10greg) [00:15:36] 10Release-Engineering-Team, 10Scap, 10Epic, 10Goal: Automate the Train - https://phabricator.wikimedia.org/T196515 (10greg) [00:17:00] 10Release-Engineering-Team (Backlog), 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10greg) next steps here? [00:17:00] 10Release-Engineering-Team (Backlog), 10Operations: Keyholder phab repo duplicate work - https://phabricator.wikimedia.org/T203003 (10greg) next steps here? [00:17:30] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Performance-Team (Radar), 10Wikimedia-production-error (Shared Build Failure): Add AntiSpoof to shared extension gate - https://phabricator.wikimedia.org/T198653 (10greg) p:05Triage>03Normal [00:17:30] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog), 10Performance-Team (Radar), 10Wikimedia-production-error (Shared Build Failure): Add AntiSpoof to shared extension gate - https://phabricator.wikimedia.org/T198653 (10greg) p:05Triage>03Normal [00:18:04] once jenkins has done its thing [00:18:04] once jenkins has done its thing [00:18:11] so are the memc* hosts actually running redis rather than memcached? or is 'redis_lock' not really redis specific? [00:18:11] so are the memc* hosts actually running redis rather than memcached? or is 'redis_lock' not really redis specific? [00:18:28] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) p:05Triage>03Normal >>! In T204252#4686261, @dduvall wrote: > @hashar, would you please triage priority and placement on the RelEng Ka... [00:18:28] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) p:05Triage>03Normal >>! In T204252#4686261, @dduvall wrote: > @hashar, would you please triage priority and placement on the RelEng Ka... [00:18:41] * bd808 thinks they are running memcached based on pstree output [00:18:41] ACTION thinks they are running memcached based on pstree output [00:18:57] (sorry, my phab spam is being noisy while ya'll are debugging) [00:18:57] (sorry, my phab spam is being noisy while ya'll are debugging) [00:19:13] I think it's not really redis specific [00:19:13] I think it's not really redis specific [00:19:38] well [00:19:38] well [00:19:47] class role::mediawiki::memcached inherits role::memcached { [00:19:47] class role::mediawiki::memcached inherits role::memcached { [00:19:47] include ::profile::redis::multidc [00:19:47] include ::profile::redis::multidc [00:20:29] hm [00:20:29] hm [00:20:52] that's mcrouter I bet [00:20:52] that's mcrouter I bet [00:21:20] looks like our memc hosts do not include redis? [00:21:21] looks like our memc hosts do not include redis? [00:21:47] 10Continuous-Integration-Config, 10Patch-For-Review: Wikibase CI: wmf-quibble-vendor-mysql-hhvm-docker job should include Scribunto - https://phabricator.wikimedia.org/T200976 (10greg) [00:21:47] 10Continuous-Integration-Config, 10Patch-For-Review: Wikibase CI: wmf-quibble-vendor-mysql-hhvm-docker job should include Scribunto - https://phabricator.wikimedia.org/T200976 (10greg) [00:21:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) [00:21:49] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban): Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) [00:21:58] I would not expect them to, no. But mcrouter is used for fronting both redis and memcached services [00:21:58] I would not expect them to, no. But mcrouter is used for fronting both redis and memcached services [00:22:04] 10Continuous-Integration-Config, 10Release-Engineering-Team: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) [00:22:05] 10Continuous-Integration-Config, 10Release-Engineering-Team: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) [00:22:32] it adds the pooled hashing magic to memcached + replicates redis changes cross-dc [00:22:32] it adds the pooled hashing magic to memcached + replicates redis changes cross-dc [00:22:47] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) actually... here. We have subtasks now. [00:22:47] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) actually... here. We have subtasks now. [00:23:05] is this something that's depended on in production but was never installed in beta? [00:23:05] is this something that's depended on in production but was never installed in beta? [00:23:11] 10Gerrit: Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10greg) [00:23:11] 10Gerrit: Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10greg) [00:23:35] 10Release-Engineering-Team (Backlog), 10Scap: Canary deployment continuation prompt should emit a timestamp to console - https://phabricator.wikimedia.org/T201635 (10greg) [00:23:35] 10Release-Engineering-Team (Backlog), 10Scap: Canary deployment continuation prompt should emit a timestamp to console - https://phabricator.wikimedia.org/T201635 (10greg) [00:24:17] Krenair: there is a ton of backscroll that I did not even try to read, but I'm confused on the desire to kill usage of redis in beta cluster [00:24:18] Krenair: there is a ton of backscroll that I did not even try to read, but I'm confused on the desire to kill usage of redis in beta cluster [00:24:32] we definately use redis for session storage in prod [00:24:32] we definately use redis for session storage in prod [00:24:48] I don't have the full picture of all the conversations that went on around it [00:24:48] I don't have the full picture of all the conversations that went on around it [00:24:51] Project beta-update-databases-eqiad build #29911: 04STILL FAILING in 2.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29911/ [00:24:51] Project beta-update-databases-eqiad build #29911: 04STILL FAILING in 2.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29911/ [00:24:57] andrewbogott should [00:24:57] andrewbogott should [00:25:12] wgSessionCacheType and wgMainStash are both set to redis [00:25:12] wgSessionCacheType and wgMainStash are both set to redis [00:25:31] Nov 20 13:30:40 Trying to find someone who will admit to owning/caring about those redis boxes. [00:25:31] Nov 20 13:42:16 So far the word on redis in _security is 'iirc changeprop in deployment-prep is not using redis at all' [00:25:31] Nov 20 13:54:30 Krenair: Giuseppe says about redis "we're not using the old jobqueue, we should remove those vms" [00:25:31] Nov 20 13:30:40 Trying to find someone who will admit to owning/caring about those redis boxes. [00:25:31] Nov 20 13:42:16 So far the word on redis in _security is 'iirc changeprop in deployment-prep is not using redis at all' [00:25:31] Nov 20 13:54:30 Krenair: Giuseppe says about redis "we're not using the old jobqueue, we should remove those vms" [00:26:03] 10Gerrit, 10Release-Engineering-Team (Kanban), 10User-Addshore: Access to create gerrit groups for WMDE-Leszek - https://phabricator.wikimedia.org/T200311 (10greg) p:05Triage>03Normal Gah, I'm not a gerrit admin, I would have JFDI now. [00:26:03] 10Gerrit, 10Release-Engineering-Team (Kanban), 10User-Addshore: Access to create gerrit groups for WMDE-Leszek - https://phabricator.wikimedia.org/T200311 (10greg) p:05Triage>03Normal Gah, I'm not a gerrit admin, I would have JFDI now. [00:26:34] 10Release-Engineering-Team (Kanban): TEC13:O3.4:Q1 Goal - Put in place Tech Debt management process for PEP - https://phabricator.wikimedia.org/T199263 (10greg) p:05Triage>03Normal a:03Jrbranaa [00:26:34] 10Release-Engineering-Team (Kanban): TEC13:O3.4:Q1 Goal - Put in place Tech Debt management process for PEP - https://phabricator.wikimedia.org/T199263 (10greg) p:05Triage>03Normal a:03Jrbranaa [00:26:41] yeah, and that's right as far a job queue and change prop go, but not session storage :) [00:26:41] yeah, and that's right as far a job queue and change prop go, but not session storage :) [00:26:52] 10Release-Engineering-Team (Kanban): TEC13:O3.4:Q1 Goal - Identify key Tech Debt areas (Platform) - https://phabricator.wikimedia.org/T199262 (10greg) p:05Triage>03Normal a:03Jrbranaa [00:26:52] 10Release-Engineering-Team (Kanban): TEC13:O3.4:Q1 Goal - Identify key Tech Debt areas (Platform) - https://phabricator.wikimedia.org/T199262 (10greg) p:05Triage>03Normal a:03Jrbranaa [00:27:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) To move sessions out of redis, this will need to change in wmf-config/InitialiseSettings.php: `lang=php 'wgSessionCacheType' => [ ⠀⠀⠀'default'... [00:27:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) To move sessions out of redis, this will need to change in wmf-config/InitialiseSettings.php: `lang=php 'wgSessionCacheType' => [ ⠀⠀⠀'default'... [00:30:36] 10Continuous-Integration-Config, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10greg) [00:30:36] 10Continuous-Integration-Config, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10greg) [00:32:01] 10Release-Engineering-Team (Watching / External), 10Scap, 10Operations, 10Patch-For-Review, and 2 others: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842 (10greg) [00:32:01] 10Release-Engineering-Team (Watching / External), 10Scap, 10Operations, 10Patch-For-Review, and 2 others: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842 (10greg) [00:32:33] 10Release-Engineering-Team (Backlog), 10Code-Health, 10Wikimedia-Incident: Look into ways to identify risky commits that are part of the weekly branch cut, either manually or through automated process - https://phabricator.wikimedia.org/T193258 (10greg) [00:32:33] 10Release-Engineering-Team (Backlog), 10Code-Health, 10Wikimedia-Incident: Look into ways to identify risky commits that are part of the weekly branch cut, either manually or through automated process - https://phabricator.wikimedia.org/T193258 (10greg) [00:33:49] Nov 20 13:44:59 mobrovac, can you tell me about deployment-redis VMs in deployment-prep? They were involved in the jobqueue so I'm guessing they're just leftovers at this point... [00:33:49] Nov 20 13:44:59 mobrovac, can you tell me about deployment-redis VMs in deployment-prep? They were involved in the jobqueue so I'm guessing they're just leftovers at this point... [00:33:49] Nov 20 13:45:05 would like confirmation before I delete them though :) [00:33:49] Nov 20 13:47:36 Go [00:33:50] Nov 20 13:48:44 Throw it away [00:33:50] Nov 20 13:45:05 would like confirmation before I delete them though :) [00:33:50] Nov 20 13:47:36 Go [00:33:50] Nov 20 13:48:44 Throw it away [00:33:50] Nov 20 13:55:45 !log deployment-prep deleting deployment-redis05 and deployment-redis06 as per Giuseppe, "we're not using the old jobqueue, we should remove those vms" [00:33:50] Nov 20 13:55:45 !log deployment-prep deleting deployment-redis05 and deployment-redis06 as per Giuseppe, "we're not using the old jobqueue, we should remove those vms" [00:33:52] from -cloud [00:33:52] from -cloud [00:34:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI setuptools tox docker test failling on cergen.git - https://phabricator.wikimedia.org/T194673 (10greg) Status? [00:34:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI setuptools tox docker test failling on cergen.git - https://phabricator.wikimedia.org/T194673 (10greg) Status? [00:34:17] I get that there was bad advice given :) [00:34:17] I get that there was bad advice given :) [00:34:49] 10Gerrit, 10Documentation: Document process to merge a reviewed but not submitted change using the gerrit web-ui - https://phabricator.wikimedia.org/T196265 (10greg) [00:34:49] 10Gerrit, 10Documentation: Document process to merge a reviewed but not submitted change using the gerrit web-ui - https://phabricator.wikimedia.org/T196265 (10greg) [00:34:49] :( Sorry. Can they be rebuilt? [00:34:49] :( Sorry. Can they be rebuilt? [00:34:59] 10Release-Engineering-Team (Backlog), 10Scap, 10Operations: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10greg) [00:34:59] 10Release-Engineering-Team (Backlog), 10Scap, 10Operations: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10greg) [00:35:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins: Enable MFA on Jenkins - https://phabricator.wikimedia.org/T198814 (10greg) [00:35:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins: Enable MFA on Jenkins - https://phabricator.wikimedia.org/T198814 (10greg) [00:35:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External): Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10greg) [00:35:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External): Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10greg) [00:35:30] horizon says both deployment-redis05 and 06 are running but ssh is not working for me [00:35:30] horizon says both deployment-redis05 and 06 are running but ssh is not working for me [00:35:55] 10Continuous-Integration-Config, 10Release-Engineering-Team: CI: Minerva PHPUnit tests should be included in shared extension gate job - https://phabricator.wikimedia.org/T202030 (10greg) [00:35:55] 10Continuous-Integration-Config, 10Release-Engineering-Team: CI: Minerva PHPUnit tests should be included in shared extension gate job - https://phabricator.wikimedia.org/T202030 (10greg) [00:35:57] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) [00:35:57] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Epic: Have dependencies of gated extensions in the gate - https://phabricator.wikimedia.org/T204252 (10greg) [00:36:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog): CI: Minerva PHPUnit tests should be included in shared extension gate job - https://phabricator.wikimedia.org/T202030 (10greg) [00:36:15] 10Continuous-Integration-Config, 10Release-Engineering-Team (Backlog): CI: Minerva PHPUnit tests should be included in shared extension gate job - https://phabricator.wikimedia.org/T202030 (10greg) [00:36:42] looks like they both had a bad time with the intial puppet runs [00:36:43] looks like they both had a bad time with the intial puppet runs [00:37:02] I can help in 15 or so [00:37:02] I can help in 15 or so [00:37:09] did someone try to recreate them? [00:37:09] did someone try to recreate them? [00:37:19] yeah I can't even get in as root [00:37:19] yeah I can't even get in as root [00:37:24] I think twentyafterfour did, yeah [00:37:24] I think twentyafterfour did, yeah [00:37:58] 2018-11-20T21:06:08.582065+00:00 host-172-16-5-192 puppet-agent[2709]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find class role::jobqueue_redis::master for deployment-redis05.deployment-prep.eqiad.wmflabs on node deployment-redis05.deployment-prep.eqiad.wmflabs [00:37:58] 2018-11-20T21:06:08.582065+00:00 host-172-16-5-192 puppet-agent[2709]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find class role::jobqueue_redis::master for deployment-redis05.deployment-prep.eqiad.wmflabs on node deployment-redis05.deployment-prep.eqiad.wmflabs [00:38:29] is that an instance mid-migration? [00:38:29] is that an instance mid-migration? [00:38:53] looks like its having some sort of identity crisis as well as puppet problems [00:38:53] looks like its having some sort of identity crisis as well as puppet problems [00:39:43] * bd808 tries to load the prefix puppet settings for these hosts [00:39:43] ACTION tries to load the prefix puppet settings for these hosts [00:48:47] ok, in prod the mc* hosts are apparently both redis and memcached instances [00:48:47] ok, in prod the mc* hosts are apparently both redis and memcached instances [00:49:05] * bd808 looks at deployment-memc prefix puppet next [00:49:05] ACTION looks at deployment-memc prefix puppet next [00:51:54] hm, I can't log in to deployment-redis05.deployment-prep.eqiad.wmflabs either [00:51:54] hm, I can't log in to deployment-redis05.deployment-prep.eqiad.wmflabs either [00:51:58] Krenair: I think you were on the right track with profile::redis::multidc. It looks to me like the deployment-prep memc boxes were never switched to be memcached+redis [00:51:58] Krenair: I think you were on the right track with profile::redis::multidc. It looks to me like the deployment-prep memc boxes were never switched to be memcached+redis [00:52:44] and instead deployment-prep was using the redis* boxes as session+job queue+whatever redis [00:52:44] and instead deployment-prep was using the redis* boxes as session+job queue+whatever redis [00:53:38] so ... we need to change deployment-memc* to use role::mediawiki::memcached instead of just role::memcached [00:53:38] so ... we need to change deployment-memc* to use role::mediawiki::memcached instead of just role::memcached [00:54:12] that should add redis to them... and then we need to figure out how to point mcrouter in the right way [00:54:12] that should add redis to them... and then we need to figure out how to point mcrouter in the right way [00:55:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) ` [00:39] bd808 tries to load the prefix puppet settings for these hosts [00:48] < bd808> ok, in prod the mc* hosts are apparently both red... [00:55:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) ` [00:39] bd808 tries to load the prefix puppet settings for these hosts [00:48] < bd808> ok, in prod the mc* hosts are apparently both red... [00:56:02] I need to go run an errand... anyone want to try applying role::mediawiki::memcached? [00:56:02] I need to go run an errand... anyone want to try applying role::mediawiki::memcached? [00:57:07] bd808: I need a little more context, want me to build fresh VMs? [00:57:07] bd808: I need a little more context, want me to build fresh VMs? [01:09:33] ok, I can't bear the suspense… ping me if there's anything I can do to help [01:09:33] ok, I can't bear the suspense… ping me if there's anything I can do to help [01:20:02] Project beta-update-databases-eqiad build #29912: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29912/ [01:20:02] Project beta-update-databases-eqiad build #29912: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29912/ [01:22:47] !log Applied role::mediawiki::memcached on deployment-memc05.deployment-prep.eqiad.wmflabs to provision redis T210030 [01:22:48] !log Applied role::mediawiki::memcached on deployment-memc05.deployment-prep.eqiad.wmflabs to provision redis T210030 [01:22:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:22:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:22:51] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [01:22:51] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [01:23:13] and now the hunt for the right hiera settings begins :) [01:23:13] and now the hunt for the right hiera settings begins :) [01:53:35] oh... I think this all needs confd/etcd magic too [01:53:35] oh... I think this all needs confd/etcd magic too [01:54:28] Would it be easier to go back to redid then migrate to that at some point? :) [01:54:28] Would it be easier to go back to redid then migrate to that at some point? :) [01:55:43] paladox: I don't have a time machine, so .... yeah. Forward is probably our best bet, but this is yet another thing that has changed a lot in prod without testing/mirroring in deployment-prep [01:55:43] paladox: I don't have a time machine, so .... yeah. Forward is probably our best bet, but this is yet another thing that has changed a lot in prod without testing/mirroring in deployment-prep [01:56:06] Oh [01:56:06] Oh [01:56:20] one of many issues here is that deployment-prep is always playing catchup on things that the core SRE team does [01:56:20] one of many issues here is that deployment-prep is always playing catchup on things that the core SRE team does [01:56:22] I meant redis [01:56:22] I meant redis [01:56:32] Auto spell checker replaced the word [01:56:32] Auto spell checker replaced the word [01:57:20] I am trying to setup redis in the way that it is used in the main prod cluster rather than try to resurrect whatever setup was cobbled together in the past [01:57:20] I am trying to setup redis in the way that it is used in the main prod cluster rather than try to resurrect whatever setup was cobbled together in the past [01:57:38] Ah ok :) [01:57:38] Ah ok :) [01:57:44] this means using role::mediawiki::memcached to provision it [01:57:44] this means using role::mediawiki::memcached to provision it [01:58:16] and that runs into lots of interesting requirements that I'm going to guess have never been used in deployment-prep [01:58:16] and that runs into lots of interesting requirements that I'm going to guess have never been used in deployment-prep [01:58:23] but I may be wrong :) [01:58:24] but I may be wrong :) [01:58:31] Heh [01:58:31] Heh [02:20:02] Project beta-update-databases-eqiad build #29913: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29913/ [02:20:08] Project beta-update-databases-eqiad build #29913: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29913/ [02:28:05] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) After applying role::mediawiki::memcached on deployment-memc05 I added some missing hiera config to the deployment-memc prefix-puppet: `lang=ya... [02:28:05] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) After applying role::mediawiki::memcached on deployment-memc05 I added some missing hiera config to the deployment-memc prefix-puppet: `lang=ya... [03:02:11] !log Repeated application of role::mediawiki::memcached on deployment-memc04, deployment-memc06, and deployment-memc07 for T210030 [03:02:11] !log Repeated application of role::mediawiki::memcached on deployment-memc04, deployment-memc06, and deployment-memc07 for T210030 [03:02:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:02:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:02:15] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [03:02:15] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [03:20:07] Project beta-update-databases-eqiad build #29914: 04STILL FAILING in 7.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29914/ [03:20:07] Project beta-update-databases-eqiad build #29914: 04STILL FAILING in 7.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29914/ [03:24:36] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) >>! In T210030#4764679, @bd808 wrote: > After applying role::mediawiki::memcached on deployment-memc05 I added some missing hiera config to the... [03:24:36] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) >>! In T210030#4764679, @bd808 wrote: > After applying role::mediawiki::memcached on deployment-memc05 I added some missing hiera config to the... [03:25:22] !log Forced puppet run on deployment-mediawiki-0[79] to pick up new redis::shards settings T210030 [03:25:22] !log Forced puppet run on deployment-mediawiki-0[79] to pick up new redis::shards settings T210030 [03:25:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:25:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:25:26] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [03:25:26] T210030: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 [03:34:21] twentyafterfour, andrewbogott, greg-g: its alive! -- https://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page [03:34:21] twentyafterfour, andrewbogott, greg-g: its alive! -- https://deployment.wikimedia.beta.wmflabs.org/wiki/Main_Page [03:35:01] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36936 bytes in 1.825 second response time [03:35:01] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36936 bytes in 1.825 second response time [03:37:29] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47950 bytes in 1.321 second response time [03:37:29] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47950 bytes in 1.321 second response time [03:39:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) The wikis are alive. :) `lang=irc [03:35] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1... [03:39:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10bd808) The wikis are alive. :) `lang=irc [03:35] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1... [03:42:48] Wooooo! [03:42:48] Wooooo! [04:17:19] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Flow\Tests\Collection\PostCollectionTest::testGetLastRevision and Flow\Tests\Api\ApiFlowLockTopicTest::testLockTopic - https://phabricator.wikimedia.org/T210043 (10D3r1ck01) p:05Triage>03Normal [04:17:19] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint): Flow\Tests\Collection\PostCollectionTest::testGetLastRevision and Flow\Tests\Api\ApiFlowLockTopicTest::testLockTopic - https://phabricator.wikimedia.org/T210043 (10D3r1ck01) p:05Triage>03Normal [04:20:02] Project beta-update-databases-eqiad build #29915: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29915/ [04:20:02] Project beta-update-databases-eqiad build #29915: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29915/ [04:20:15] bd808: yay! [04:20:15] bd808: yay! [04:20:53] thanks all, bd808 twentyafterfour Krenair andrewbogott [04:20:54] thanks all, bd808 twentyafterfour Krenair andrewbogott [04:36:38] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), 10Patch-For-Review: Flow\Tests\Collection\PostCollectionTest::testGetLastRevision and Flow\Tests\Api\ApiFlowLockTopicTest::testLockTopic - https://phabricator.wikimedia.org/T210043 (10D3r1ck01) a:03... [04:36:38] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), 10Patch-For-Review: Flow\Tests\Collection\PostCollectionTest::testGetLastRevision and Flow\Tests\Api\ApiFlowLockTopicTest::testLockTopic - https://phabricator.wikimedia.org/T210043 (10D3r1ck01) a:03... [04:37:01] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), and 2 others: Flow\Tests\Collection\PostCollectionTest::testGetLastRevision and Flow\Tests\Api\ApiFlowLockTopicTest::testLockTopic - https://phabricator.wikimedia.org/T210043 (10D3r1ck01) [04:37:01] 10Continuous-Integration-Config, 10StructuredDiscussions, 10User-kostajh, 10Growth-Team (Current Sprint), and 2 others: Flow\Tests\Collection\PostCollectionTest::testGetLastRevision and Flow\Tests\Api\ApiFlowLockTopicTest::testLockTopic - https://phabricator.wikimedia.org/T210043 (10D3r1ck01) [05:20:06] Project beta-update-databases-eqiad build #29916: 04STILL FAILING in 6.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29916/ [05:20:07] Project beta-update-databases-eqiad build #29916: 04STILL FAILING in 6.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29916/ [06:20:02] Project beta-update-databases-eqiad build #29917: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29917/ [06:20:02] Project beta-update-databases-eqiad build #29917: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29917/ [07:02:05] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Krishna_Chaitanya_Velaga) @Aklapper Yes, I would need to create more tasks projects in future, for a range of outreach activities. I frequently organise activities, and I see dashboa... [07:02:05] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Krishna_Chaitanya_Velaga) @Aklapper Yes, I would need to create more tasks projects in future, for a range of outreach activities. I frequently organise activities, and I see dashboa... [07:20:02] Project beta-update-databases-eqiad build #29918: 04STILL FAILING in 2.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29918/ [07:20:02] Project beta-update-databases-eqiad build #29918: 04STILL FAILING in 2.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29918/ [07:42:10] (03PS2) 10Zoranzoki21: Fix description about basic check test [integration/config] - 10https://gerrit.wikimedia.org/r/471096 [07:42:10] (03PS2) 10Zoranzoki21: Fix description about basic check test [integration/config] - 10https://gerrit.wikimedia.org/r/471096 [07:47:45] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [07:47:45] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [08:10:41] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [08:10:41] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [08:12:35] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) Big re-arrangement of the checklist (sorry, I couldn't sleep); hopefully an improvement. I left some things un-checked that you may or may not have already done. [08:12:35] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) Big re-arrangement of the checklist (sorry, I couldn't sleep); hopefully an improvement. I left some things un-checked that you may or may not have already done. [08:19:42] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [08:19:43] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [08:20:02] Project beta-update-databases-eqiad build #29919: 04STILL FAILING in 2.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29919/ [08:20:02] Project beta-update-databases-eqiad build #29919: 04STILL FAILING in 2.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29919/ [08:20:43] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) > [X] Ask Greg to setup a team [[ https://en.wikipedia.org/wiki/Key_signing_party | signing party ]]. > [X] [[ https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Onboarding#People_to_... [08:20:43] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) > [X] Ask Greg to setup a team [[ https://en.wikipedia.org/wiki/Key_signing_party | signing party ]]. > [X] [[ https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Onboarding#People_to_... [09:13:57] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Dvorapa) No, we need your help! [09:13:57] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Dvorapa) No, we need your help! [09:20:10] Project beta-update-databases-eqiad build #29920: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29920/ [09:20:10] Project beta-update-databases-eqiad build #29920: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29920/ [10:24:16] Project beta-update-databases-eqiad build #29921: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29921/ [10:24:16] Project beta-update-databases-eqiad build #29921: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29921/ [10:29:02] 10MediaWiki-Releasing, 10Growth-Team, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650 (10Seb35) Given this decision, should we remove the tag #MW-1.32-release in the unitary tasks and add #MW-1.... [10:29:02] 10MediaWiki-Releasing, 10Growth-Team, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650 (10Seb35) Given this decision, should we remove the tag #MW-1.32-release in the unitary tasks and add #MW-1.... [10:39:04] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10hashar) >>! In T183860#4683354, @Dvorapa wrote: > I'm not sure, why I added RepoAds, sorry. The problem is t... [10:39:04] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10hashar) >>! In T183860#4683354, @Dvorapa wrote: > I'm not sure, why I added RepoAds, sorry. The problem is t... [10:41:10] 10Release-Engineering-Team (Watching / External), 10Pywikibot: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306 (10hashar) There is some documentation at https://www.appveyor.com/docs/team-setup/ Seems the AppVeyor accounts/team can be tied to the GitHub o... [10:41:10] 10Release-Engineering-Team (Watching / External), 10Pywikibot: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306 (10hashar) There is some documentation at https://www.appveyor.com/docs/team-setup/ Seems the AppVeyor accounts/team can be tied to the GitHub o... [10:43:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI setuptools tox docker test failling on cergen.git - https://phabricator.wikimedia.org/T194673 (10hashar) 05Open>03Resolved a:03hashar Fixed by setting `setuptools_scm < 2.0.0` in setup.py setup_requires: https://gerrit.wikimedia.org... [10:43:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI setuptools tox docker test failling on cergen.git - https://phabricator.wikimedia.org/T194673 (10hashar) 05Open>03Resolved a:03hashar Fixed by setting `setuptools_scm < 2.0.0` in setup.py setup_requires: https://gerrit.wikimedia.org... [11:00:35] (03PS1) 10Ema: Build prometheus-php-fpm-exporter with '-non-voting' [integration/config] - 10https://gerrit.wikimedia.org/r/475081 (https://phabricator.wikimedia.org/T209573) [11:00:35] (03PS1) 10Ema: Build prometheus-php-fpm-exporter with '-non-voting' [integration/config] - 10https://gerrit.wikimedia.org/r/475081 (https://phabricator.wikimedia.org/T209573) [11:05:35] (03CR) 10Addshore: [C: 032] readme: how to reproduce a CI build [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [11:05:35] (03CR) 10Addshore: [C: 032] readme: how to reproduce a CI build [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [11:06:30] (03Merged) 10jenkins-bot: readme: how to reproduce a CI build [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [11:06:30] (03Merged) 10jenkins-bot: readme: how to reproduce a CI build [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [11:06:59] (03CR) 10jenkins-bot: readme: how to reproduce a CI build [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [11:06:59] (03CR) 10jenkins-bot: readme: how to reproduce a CI build [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [11:20:11] Project beta-update-databases-eqiad build #29922: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29922/ [11:20:11] Project beta-update-databases-eqiad build #29922: 04STILL FAILING in 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29922/ [11:22:19] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: debianize docker-registry 2.7.0-rc0 and upload in stretch-wikimedia - https://phabricator.wikimedia.org/T210071 (10fselles) p:05Triage>03Normal [11:22:19] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: debianize docker-registry 2.7.0-rc0 and upload in stretch-wikimedia - https://phabricator.wikimedia.org/T210071 (10fselles) p:05Triage>03Normal [11:59:07] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: set up a test node with new version, redis as cache, a new swift container and export metrics over graphana - https://phabricator.wikimedia.org/T210076 (10fselles) p:05Triage>03Normal [11:59:07] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: set up a test node with new version, redis as cache, a new swift container and export metrics over graphana - https://phabricator.wikimedia.org/T210076 (10fselles) p:05Triage>03Normal [12:07:16] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Dvorapa) >>! In T183860#4765323, @hashar wrote: >>>! In T183860#4683354, @Dvorapa wrote: >> I'm not sure, wh... [12:07:16] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Dvorapa) >>! In T183860#4765323, @hashar wrote: >>>! In T183860#4683354, @Dvorapa wrote: >> I'm not sure, wh... [12:10:28] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Dvorapa) [12:10:28] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Dvorapa) [12:20:02] Project beta-update-databases-eqiad build #29923: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29923/ [12:20:02] Project beta-update-databases-eqiad build #29923: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29923/ [12:26:52] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Krenair) There is a request open to install AppVeyor on some of the wikimedia github repositories. Is it sup... [12:26:52] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot, 10Pywikibot-tests: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860 (10Krenair) There is a request open to install AppVeyor on some of the wikimedia github repositories. Is it sup... [13:20:06] Project beta-update-databases-eqiad build #29924: 04STILL FAILING in 6.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29924/ [13:20:06] Project beta-update-databases-eqiad build #29924: 04STILL FAILING in 6.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29924/ [14:01:57] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) >>! In T199133#4732915, @Ladsgroup wrote: > Just saying, the "ORES" service and backend of the ORES extension i... [14:01:57] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10zeljkofilipin) >>! In T199133#4732915, @Ladsgroup wrote: > Just saying, the "ORES" service and backend of the ORES extension i... [14:19:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10fgiunchedi) This is working, though we'll need to adjust the "level" field: ` { "_index":... [14:19:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10fgiunchedi) This is working, though we'll need to adjust the "level" field: ` { "_index":... [14:20:06] Project beta-update-databases-eqiad build #29925: 04STILL FAILING in 6.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29925/ [14:20:06] Project beta-update-databases-eqiad build #29925: 04STILL FAILING in 6.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29925/ [14:51:39] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] Fix description about basic check test [integration/config] - 10https://gerrit.wikimedia.org/r/471096 (owner: 10Zoranzoki21) [14:51:39] (03CR) 10Thiemo Kreuz (WMDE): [C: 031] Fix description about basic check test [integration/config] - 10https://gerrit.wikimedia.org/r/471096 (owner: 10Zoranzoki21) [15:06:41] 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.33-release: Bundle Scribunto extension with MediaWiki - https://phabricator.wikimedia.org/T191737 (10CCicalese_WMF) [15:06:41] 10MediaWiki-Releasing, 10MediaWiki-extensions-Scribunto, 10MW-1.33-release: Bundle Scribunto extension with MediaWiki - https://phabricator.wikimedia.org/T191737 (10CCicalese_WMF) [15:06:55] 10MediaWiki-Releasing, 10Growth-Team, 10Notifications, 10MW-1.33-release: Bundle Echo extension with MediaWiki - https://phabricator.wikimedia.org/T191738 (10CCicalese_WMF) [15:06:55] 10MediaWiki-Releasing, 10Growth-Team, 10Notifications, 10MW-1.33-release: Bundle Echo extension with MediaWiki - https://phabricator.wikimedia.org/T191738 (10CCicalese_WMF) [15:07:52] 10MediaWiki-Releasing, 10MobileFrontend, 10MW-1.33-release, 10Readers-Web-Backlog (Tracking): Bundle MobileFrontend extension with MediaWiki - https://phabricator.wikimedia.org/T191734 (10CCicalese_WMF) [15:07:52] 10MediaWiki-Releasing, 10MobileFrontend, 10MW-1.33-release, 10Readers-Web-Backlog (Tracking): Bundle MobileFrontend extension with MediaWiki - https://phabricator.wikimedia.org/T191734 (10CCicalese_WMF) [15:07:58] 10MediaWiki-Releasing, 10Growth-Team, 10Thanks, 10MW-1.33-release: Bundle Thanks extension with MediaWiki - https://phabricator.wikimedia.org/T191739 (10CCicalese_WMF) [15:07:58] 10MediaWiki-Releasing, 10Growth-Team, 10Thanks, 10MW-1.33-release: Bundle Thanks extension with MediaWiki - https://phabricator.wikimedia.org/T191739 (10CCicalese_WMF) [15:08:04] 10MediaWiki-Releasing, 10AbuseFilter, 10MW-1.33-release, 10Patch-For-Review, 10User-Daimona: Bundle AbuseFilter extension with MediaWiki - https://phabricator.wikimedia.org/T191740 (10CCicalese_WMF) [15:08:04] 10MediaWiki-Releasing, 10AbuseFilter, 10MW-1.33-release, 10Patch-For-Review, 10User-Daimona: Bundle AbuseFilter extension with MediaWiki - https://phabricator.wikimedia.org/T191740 (10CCicalese_WMF) [15:08:07] 10MediaWiki-Releasing, 10MinervaNeue, 10MW-1.33-release, 10Readers-Web-Backlog (Tracking): Bundle Minerva Neue skin with MediaWiki - https://phabricator.wikimedia.org/T191743 (10CCicalese_WMF) [15:08:07] 10MediaWiki-Releasing, 10MinervaNeue, 10MW-1.33-release, 10Readers-Web-Backlog (Tracking): Bundle Minerva Neue skin with MediaWiki - https://phabricator.wikimedia.org/T191743 (10CCicalese_WMF) [15:08:17] 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.33-release: Bundle LoginNotify extension with MediaWiki - https://phabricator.wikimedia.org/T191746 (10CCicalese_WMF) [15:08:17] 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.33-release: Bundle LoginNotify extension with MediaWiki - https://phabricator.wikimedia.org/T191746 (10CCicalese_WMF) [15:08:44] 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.33-release: Bundle AntiSpoof extension with MediaWiki - https://phabricator.wikimedia.org/T191736 (10CCicalese_WMF) [15:08:44] 10MediaWiki-Releasing, 10AntiSpoof, 10MW-1.33-release: Bundle AntiSpoof extension with MediaWiki - https://phabricator.wikimedia.org/T191736 (10CCicalese_WMF) [15:10:24] 10MediaWiki-Releasing, 10Growth-Team, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650 (10CCicalese_WMF) Yes. I just made that change. [15:10:24] 10MediaWiki-Releasing, 10Growth-Team, 10MediaWiki-Installer, 10Epic, 10MW-1.32-release: Expand the set of bundled extensions and skins in MediaWiki 1.32 - https://phabricator.wikimedia.org/T196650 (10CCicalese_WMF) Yes. I just made that change. [15:20:02] Project beta-update-databases-eqiad build #29926: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29926/ [15:20:02] Project beta-update-databases-eqiad build #29926: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29926/ [15:22:01] PROBLEM - Host integration-slave-jessie-android is DOWN: CRITICAL - Host Unreachable (10.68.19.239) [15:22:01] PROBLEM - Host integration-slave-jessie-android is DOWN: CRITICAL - Host Unreachable (10.68.19.239) [15:26:32] PROBLEM - Host integration-slave-docker-1043 is DOWN: CRITICAL - Host Unreachable (10.68.22.120) [15:26:32] PROBLEM - Host integration-slave-docker-1043 is DOWN: CRITICAL - Host Unreachable (10.68.22.120) [15:53:11] ^^ I'm moving those two, they'll be back in an hour or so [15:53:11] ^^ I'm moving those two, they'll be back in an hour or so [16:08:52] 10Beta-Cluster-Infrastructure: UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10kostajh) [16:08:52] 10Beta-Cluster-Infrastructure: UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10kostajh) [16:15:58] 10Beta-Cluster-Infrastructure: UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) `lang=bash krenair@deployment-deploy01:~$ ./deployment-tin-home/get-words-lists.sh % Total % Received % Xferd Average Speed Time... [16:15:58] 10Beta-Cluster-Infrastructure: UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) `lang=bash krenair@deployment-deploy01:~$ ./deployment-tin-home/get-words-lists.sh % Total % Received % Xferd Average Speed Time... [16:19:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10Ladsgroup) >>! In T199133#4765747, @zeljkofilipin wrote: > Thanks, but [[ https://www.mediawiki.org/wiki/Developers/Maintainer... [16:19:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Find top 15 target projects that could use Selenium tests to prevent incidents - https://phabricator.wikimedia.org/T199133 (10Ladsgroup) >>! In T199133#4765747, @zeljkofilipin wrote: > Thanks, but [[ https://www.mediawiki.org/wiki/Developers/Maintainer... [16:20:02] Project beta-update-databases-eqiad build #29927: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29927/ [16:20:02] Project beta-update-databases-eqiad build #29927: 04STILL FAILING in 1.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29927/ [16:29:19] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Aklapper) [16:29:19] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Aklapper) [17:20:07] Project beta-update-databases-eqiad build #29928: 04STILL FAILING in 6.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29928/ [17:20:07] Project beta-update-databases-eqiad build #29928: 04STILL FAILING in 6.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29928/ [18:17:17] paladox: voice set on new team members :) [18:17:17] paladox: voice set on new team members :) [18:17:22] :) [18:17:22] :) [18:17:25] thanks! [18:17:25] thanks! [18:20:03] Project beta-update-databases-eqiad build #29929: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29929/ [18:20:03] Project beta-update-databases-eqiad build #29929: 04STILL FAILING in 2.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29929/ [18:20:33] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [18:20:33] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10greg) [18:28:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T206662 (10greg) a:03zeljkofilipin [18:28:26] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T206662 (10greg) a:03zeljkofilipin [18:28:32] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10RobH) So this requires an entry in the admins module, and you already have a pending entry for your shell request. Can this wait and merge with that, or does it need to h... [18:28:32] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10RobH) So this requires an entry in the admins module, and you already have a pending entry for your shell request. Can this wait and merge with that, or does it need to h... [18:29:56] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10greg) It can wait. Thanks! [18:29:56] 10Release-Engineering-Team (Kanban), 10LDAP-Access-Requests: LDAP requests for Jeena Huneidi - https://phabricator.wikimedia.org/T210028 (10greg) It can wait. Thanks! [19:04:01] subbu: Hej, could anyone in the Parser team look at https://phabricator.wikimedia.org/T209236 ? I'm a bit worried if other HTML entities might also affect URL parameters (though cannot come up with another obvious example) [19:04:01] subbu: Hej, could anyone in the Parser team look at https://phabricator.wikimedia.org/T209236 ? I'm a bit worried if other HTML entities might also affect URL parameters (though cannot come up with another obvious example) [19:20:07] Project beta-update-databases-eqiad build #29930: 04STILL FAILING in 7.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29930/ [19:20:07] Project beta-update-databases-eqiad build #29930: 04STILL FAILING in 7.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29930/ [19:30:55] hey, is blubber working fine with python? I made a blubber file for a python app and get this error at python step [19:30:55] hey, is blubber working fine with python? I made a blubber file for a python app and get this error at python step [19:31:03] https://www.irccloud.com/pastebin/YgF4a69w/ [19:31:03] https://www.irccloud.com/pastebin/YgF4a69w/ [19:36:45] Amir1: hrm, wonder if the ca-certificates package is needed? [19:36:45] Amir1: hrm, wonder if the ca-certificates package is needed? [19:37:03] It's very likely, let me try [19:37:03] It's very likely, let me try [19:37:10] we don't have any python project using blubber yet, although it works in theory :) [19:37:10] we don't have any python project using blubber yet, although it works in theory :) [19:38:06] Get ready to have the first one :D This is a very small python app, I want to do it to get familiar when I'll be doing it for ORES [19:38:06] Get ready to have the first one :D This is a very small python app, I want to do it to get familiar when I'll be doing it for ORES [19:38:29] I was wondering if we can productionize it :D [19:38:29] I was wondering if we can productionize it :D [19:38:46] awesome! [19:38:46] awesome! [19:40:11] One thing is that in prod we can't talk to pypi.python.org, that's why ores ships all of its wheels [19:40:11] One thing is that in prod we can't talk to pypi.python.org, that's why ores ships all of its wheels [19:40:20] most of the "shoulds" and "musts" are in https://wikitech.wikimedia.org/wiki/Streamlined_Service_Delivery_Design the tl;dr is blubber test and production entrypoint, and has OpenAPI spec that passes service checker. [19:40:20] most of the "shoulds" and "musts" are in https://wikitech.wikimedia.org/wiki/Streamlined_Service_Delivery_Design the tl;dr is blubber test and production entrypoint, and has OpenAPI spec that passes service checker. [19:40:34] s/entrypoint/variant [19:40:34] s/entrypoint/variant [19:41:38] hmm, nice [19:41:38] hmm, nice [19:42:50] the idea is that we'll build all the dependencies and then use multi-container dockerfiles to copy artifacts (like wheels) over the prod image, test that image, and then promote that image through to production (after it passes tests) [19:42:50] the idea is that we'll build all the dependencies and then use multi-container dockerfiles to copy artifacts (like wheels) over the prod image, test that image, and then promote that image through to production (after it passes tests) [19:50:03] I see [19:50:03] I see [19:57:14] https://wikitech.wikimedia.org/wiki/Streamlined_Service_Delivery_Design/research#/media/File:Continous-development-seq.png [19:57:14] https://wikitech.wikimedia.org/wiki/Streamlined_Service_Delivery_Design/research#/media/File:Continous-development-seq.png [19:58:03] thcipriani: It's actually looks awesome to me, the only thing is the need for a RelEng for deploying every patch, that's not feasible as it's mentioned already [19:58:03] thcipriani: It's actually looks awesome to me, the only thing is the need for a RelEng for deploying every patch, that's not feasible as it's mentioned already [19:58:25] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [19:58:25] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [20:01:07] Anyway, keep up the great work, I'll be bugging a lot soon [20:01:07] Anyway, keep up the great work, I'll be bugging a lot soon [20:01:35] Amir1: the current plan is to have deployments for services stay the same in terms of roles and responsibilities (i.e., service owners deploy their own code). The underlying software/commands will be different than scap, but roles are the same. [20:01:35] Amir1: the current plan is to have deployments for services stay the same in terms of roles and responsibilities (i.e., service owners deploy their own code). The underlying software/commands will be different than scap, but roles are the same. [20:02:47] currently deployment is handled via a script called "scap-helm" :) [20:02:47] currently deployment is handled via a script called "scap-helm" :) [20:03:47] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/kubernetes/scap-helm.sh [20:03:47] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/kubernetes/scap-helm.sh [20:04:02] which exists on the deployment servers [20:04:02] which exists on the deployment servers [20:04:30] Yeah, I have seen SAL and deployments for Mathoid before [20:04:30] Yeah, I have seen SAL and deployments for Mathoid before [20:04:35] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [20:04:35] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [20:05:37] how do I tell who has 2fa in phab or not? [20:05:38] how do I tell who has 2fa in phab or not? [20:08:11] figure it out :) [20:08:11] figure it out :) [20:11:45] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10chasemp) [20:11:45] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10chasemp) [20:13:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10mmodell) 05Open>03Resolved a:03bd808 [20:13:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10mmodell) 05Open>03Resolved a:03bd808 [20:13:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10mmodell) [20:13:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10mmodell) [20:13:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10mmodell) thanks for the help everyone! [20:13:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: RedisBagOStuff is broken on beta - https://phabricator.wikimedia.org/T210030 (10mmodell) thanks for the help everyone! [20:15:54] greg-g: For Phab admins, the "Has MFA" dropdown field in the query interface of https://phabricator.wikimedia.org/people/ [20:15:54] greg-g: For Phab admins, the "Has MFA" dropdown field in the query interface of https://phabricator.wikimedia.org/people/ [20:15:59] (but you might know by now) [20:15:59] (but you might know by now) [20:16:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) Jenkins emit the log to stdout or stderr which are captured by systemd and relayed t... [20:16:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) Jenkins emit the log to stdout or stderr which are captured by systemd and relayed t... [20:16:28] andre__: yup, I got there finally :) [20:16:28] andre__: yup, I got there finally :) [20:18:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) [20:18:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) [20:19:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) [20:19:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins daemon logs to logstash - https://phabricator.wikimedia.org/T143733 (10hashar) [20:20:02] Project beta-update-databases-eqiad build #29931: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29931/ [20:20:02] Project beta-update-databases-eqiad build #29931: 04STILL FAILING in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29931/ [20:30:36] (03PS2) 10Thcipriani: [WIP] Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 [20:30:36] (03PS2) 10Thcipriani: [WIP] Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 [20:31:38] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [20:31:38] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [20:34:54] (03PS3) 10Thcipriani: deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 [20:34:54] (03PS3) 10Thcipriani: deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 [20:35:38] (03CR) 10jerkins-bot: [V: 04-1] deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [20:35:38] (03CR) 10jerkins-bot: [V: 04-1] deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 (owner: 10Thcipriani) [20:37:12] (03PS4) 10Thcipriani: deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 [20:37:12] (03PS4) 10Thcipriani: deploy notes: Add train committers to blocker task [tools/release] - 10https://gerrit.wikimedia.org/r/471867 [20:38:41] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: set up a test node with new version, Redis as cache, a new Swift container and export metrics over Fraphana - https://phabricator.wikimedia.org/T210076 (10hashar) [20:38:41] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Kubernetes: set up a test node with new version, Redis as cache, a new Swift container and export metrics over Fraphana - https://phabricator.wikimedia.org/T210076 (10hashar) [20:49:37] !log gerrit: added WMDE-leszek to group "Gerrit Managers" https://gerrit.wikimedia.org/r/#/admin/groups/119,members | T200311 [20:49:37] !log gerrit: added WMDE-leszek to group "Gerrit Managers" https://gerrit.wikimedia.org/r/#/admin/groups/119,members | T200311 [20:49:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:49:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:49:41] T200311: Access to create gerrit groups for WMDE-Leszek - https://phabricator.wikimedia.org/T200311 [20:49:41] T200311: Access to create gerrit groups for WMDE-Leszek - https://phabricator.wikimedia.org/T200311 [20:54:32] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [20:54:32] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [21:05:49] 10Gerrit, 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10hashar) We have the plugin installed: https://gerrit.wikimedia.org/r/plugins/reviewers-by-blame/Documentation/index.html as well as the reviewer one https://gerrit... [21:05:49] 10Gerrit, 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10hashar) We have the plugin installed: https://gerrit.wikimedia.org/r/plugins/reviewers-by-blame/Documentation/index.html as well as the reviewer one https://gerrit... [21:14:09] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [21:14:09] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [21:16:16] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [21:16:16] 10Release-Engineering-Team (Kanban): Onboarding Jeena Huneidi - https://phabricator.wikimedia.org/T209722 (10jeena) [21:20:02] Project beta-update-databases-eqiad build #29932: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29932/ [21:20:02] Project beta-update-databases-eqiad build #29932: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29932/ [21:44:14] !log Changed beta.wmflabs.org and *.beta.wmflabs.org A records from 208.80.155.135 (deployment-cache-text04) to 185.15.56.36 (deployment-cache-text05) [21:44:14] !log Changed beta.wmflabs.org and *.beta.wmflabs.org A records from 208.80.155.135 (deployment-cache-text04) to 185.15.56.36 (deployment-cache-text05) [21:44:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:44:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:46:55] seems to be working fine [21:46:55] seems to be working fine [21:47:27] (this is with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475150/ + a hiera change on the instance to fix varnish frontend -> backend traffic) [21:47:27] (this is with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/475150/ + a hiera change on the instance to fix varnish frontend -> backend traffic) [21:47:34] 10Gerrit, 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10hashar) The plugin is enabled. On [[ https://gerrit.wikimedia.org/r/#/admin/projects/All-Projects | All-Projects ]] there are some configuration bits available: *... [21:47:34] 10Gerrit, 10Developer-Wishlist (2017), 10Patch-For-Review: Enable Gerrit reviewers-by-blame plugin - https://phabricator.wikimedia.org/T101131 (10hashar) The plugin is enabled. On [[ https://gerrit.wikimedia.org/r/#/admin/projects/All-Projects | All-Projects ]] there are some configuration bits available: *... [21:54:45] 10Gerrit: Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10hashar) p:05Triage>03Low Seems like the plugin had a lot of activity recently. Might be worth looking at again. I had a few cases for renaming projects, though it is set in stone that we can not rename them so we have ada... [21:54:45] 10Gerrit: Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10hashar) p:05Triage>03Low Seems like the plugin had a lot of activity recently. Might be worth looking at again. I had a few cases for renaming projects, though it is set in stone that we can not rename them so we have ada... [21:56:09] I am monitoring `sudo varnishlog -q 'ReqURL !~ "^/check$"'` on the old host [21:56:09] I am monitoring `sudo varnishlog -q 'ReqURL !~ "^/check$"'` on the old host [21:56:46] if its empty for a while I'll shut down the instance [21:56:47] if its empty for a while I'll shut down the instance [22:01:39] (03CR) 10Hashar: [C: 032] "Thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/471096 (owner: 10Zoranzoki21) [22:01:39] (03CR) 10Hashar: [C: 032] "Thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/471096 (owner: 10Zoranzoki21) [22:02:46] (03CR) 10Hashar: [C: 032] "Easy :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/475081 (https://phabricator.wikimedia.org/T209573) (owner: 10Ema) [22:02:46] (03CR) 10Hashar: [C: 032] "Easy :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/475081 (https://phabricator.wikimedia.org/T209573) (owner: 10Ema) [22:04:56] (03Merged) 10jenkins-bot: Fix description about basic check test [integration/config] - 10https://gerrit.wikimedia.org/r/471096 (owner: 10Zoranzoki21) [22:04:56] (03Merged) 10jenkins-bot: Fix description about basic check test [integration/config] - 10https://gerrit.wikimedia.org/r/471096 (owner: 10Zoranzoki21) [22:06:06] (03Merged) 10jenkins-bot: Build prometheus-php-fpm-exporter with '-non-voting' [integration/config] - 10https://gerrit.wikimedia.org/r/475081 (https://phabricator.wikimedia.org/T209573) (owner: 10Ema) [22:06:06] (03Merged) 10jenkins-bot: Build prometheus-php-fpm-exporter with '-non-voting' [integration/config] - 10https://gerrit.wikimedia.org/r/475081 (https://phabricator.wikimedia.org/T209573) (owner: 10Ema) [22:13:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) I just moved the A records over from deployment-cache-text04 to deployment-cache-text05. It seems to... [22:13:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Cloud-Services, 10Epic, 10Patch-For-Review: Migrate deployment-prep to eqiad1 - https://phabricator.wikimedia.org/T208101 (10Krenair) I just moved the A records over from deployment-cache-text04 to deployment-cache-text05. It seems to... [22:20:07] Project beta-update-databases-eqiad build #29933: 04STILL FAILING in 7.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29933/ [22:20:07] Project beta-update-databases-eqiad build #29933: 04STILL FAILING in 7.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29933/ [22:24:36] 10Phabricator (Upstream), 10UniversalLanguageSelector, 10Patch-For-Review, 10Upstream: ULS Suggested languages is cutting of language names vertically - https://phabricator.wikimedia.org/T210111 (10Volker_E) Patch provided upstream at https://github.com/wikimedia/jquery.uls/pull/328 [22:24:36] 10Phabricator (Upstream), 10UniversalLanguageSelector, 10Patch-For-Review, 10Upstream: ULS Suggested languages is cutting of language names vertically - https://phabricator.wikimedia.org/T210111 (10Volker_E) Patch provided upstream at https://github.com/wikimedia/jquery.uls/pull/328 [22:31:23] found another thing missing [22:31:24] found another thing missing [22:31:31] the ms-be servers store IPs of each other [22:31:31] the ms-be servers store IPs of each other [22:31:37] so: [22:31:37] so: [22:31:37] krenair@deployment-ms-be03:~$ sudo swift-recon --human-readable --diskusage [22:31:37] =============================================================================== [22:31:37] --> Starting reconnaissance on 2 hosts (object) [22:31:38] krenair@deployment-ms-be03:~$ sudo swift-recon --human-readable --diskusage [22:31:38] =============================================================================== [22:31:38] =============================================================================== [22:31:38] --> Starting reconnaissance on 2 hosts (object) [22:31:38] =============================================================================== [22:31:39] [2018-11-21 22:30:43] Checking disk usage now [22:31:39] [2018-11-21 22:30:43] Checking disk usage now [22:31:40] -> http://10.68.22.125:6000/recon/diskusage: [22:31:40] -> http://10.68.22.125:6000/recon/diskusage: [22:31:42] -> http://10.68.16.139:6000/recon/diskusage: [22:31:42] -> http://10.68.16.139:6000/recon/diskusage: [22:31:44] No hosts returned valid data. [22:31:44] No hosts returned valid data. [22:31:50] =============================================================================== [22:31:50] =============================================================================== [22:32:43] you will have to update the ring files [22:32:43] you will have to update the ring files [22:33:18] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) `krenair@deployment-ms-be03:~$ sudo swift-recon --human-readable --diskusage =======================... [22:33:18] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) `krenair@deployment-ms-be03:~$ sudo swift-recon --human-readable --diskusage =======================... [22:51:19] paladox, done, I think [22:51:19] paladox, done, I think [22:51:50] oh nope [22:51:50] oh nope [22:51:55] only on -ms-be03 [22:51:55] only on -ms-be03 [22:51:59] gotta do it again on -ms-be04 [22:51:59] gotta do it again on -ms-be04 [22:52:34] oh and -ms-fe02. lovely. [22:52:34] oh and -ms-fe02. lovely. [22:54:41] heh [22:54:41] heh [23:05:15] paladox, yay, think I got it [23:05:15] paladox, yay, think I got it [23:05:29] :) [23:05:29] :) [23:07:15] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) After running what must have been over a hundred different swift-ring-builder commands across -ms-b... [23:07:15] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) After running what must have been over a hundred different swift-ring-builder commands across -ms-b... [23:09:42] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) Captcha generation process now running. [23:09:42] 10Beta-Cluster-Infrastructure, 10ConfirmEdit (CAPTCHA extension): UnderflowException when creating an account, FancyCaptcha.php: Ran out of captcha images - https://phabricator.wikimedia.org/T210092 (10Krenair) Captcha generation process now running. [23:11:16] Krenair hundred?! [23:11:16] Krenair hundred?! [23:11:36] paladox, well, not a hundred on each host [23:11:36] paladox, well, not a hundred on each host [23:11:40] but still quite a few [23:11:40] but still quite a few [23:11:42] heh [23:11:42] heh [23:11:59] maybe 30-40 on each host [23:11:59] maybe 30-40 on each host [23:12:15] granted I may have been doing it an inefficient way [23:12:15] granted I may have been doing it an inefficient way [23:20:02] Project beta-update-databases-eqiad build #29934: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29934/ [23:20:02] Project beta-update-databases-eqiad build #29934: 04STILL FAILING in 1.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/29934/ [23:22:23] ah i see [23:22:23] ah i see [23:22:39] Krenair how many entries are there in the rings? [23:22:39] Krenair how many entries are there in the rings? [23:22:44] 30-40 references? [23:22:44] 30-40 references? [23:23:06] 4 rings, each having 3 existing entries to be replaced with 2 new entries [23:23:06] 4 rings, each having 3 existing entries to be replaced with 2 new entries [23:26:57] oh [23:26:57] oh [23:47:22] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10Smalyshev) The build is still failing. Is there somebody that should do something, and if so, who and what? [23:47:22] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10Smalyshev) The build is still failing. Is there somebody that should do something, and if so, who and what? [23:47:37] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10Smalyshev) p:05Triage>03High [23:47:37] 10Continuous-Integration-Infrastructure (shipyard), 10Wikidata, 10Wikidata Query UI: WDQS GUI build fails on CI - https://phabricator.wikimedia.org/T209776 (10Smalyshev) p:05Triage>03High