[00:00:32] 10Release-Engineering-Team, 10MediaWiki-JobRunner, 10Performance-Team: Investigate 30x increase in Jobrunner errors - https://phabricator.wikimedia.org/T171371#3480177 (10Krinkle) [00:17:34] !log Testing job insertion on beta cluster from deployment-tin triggers PHP Notice: Undefined index: uuid in EventBus/JobQueueEventBus.php:102, PHP Notice: Undefined index: sha1 in EventBus/JobQueueEventBus.php:99 [00:17:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:56:21] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3480257 (10Dzahn) 17:27 < mutante> !log bromine sudo -E reprepro clearvanished to deleted unused precise-mediawik... [01:34:35] 10Release-Engineering-Team (Kanban), 10Operations, 10vm-requests, 10Patch-For-Review: setup releases2001.codfw.wmnet - https://phabricator.wikimedia.org/T171917#3480300 (10Dzahn) ``` [!!] Install the GRUB boot loader on a hard disk ├┐ │... [02:12:41] 10MediaWiki-Codesniffer: Check @covers against autoloader with codesniffer - https://phabricator.wikimedia.org/T171899#3480321 (10Legoktm) Hmmm, isn't this something that PHPUnit should do? +cc @Krinkle [02:17:17] (03CR) 10Legoktm: "Is this still necessary? AIUI with the new Gerrit version, if someone has CR +2, and then jenkins runs and fails, if they comment anything" [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar) [03:42:20] (03CR) 10Krinkle: "I know Wikibugs sees it that way, but afaik this is not the case for Zuul somehow. I'm still having to remove my vote and re-+2 for it to " [integration/config] - 10https://gerrit.wikimedia.org/r/368154 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar) [03:52:47] 10MediaWiki-Codesniffer: Check @covers against autoloader with codesniffer - https://phabricator.wikimedia.org/T171899#3480378 (10Krinkle) @Legoktm Indeed. For PHPCS to statically be aware of all classes seems unrealistic at this point. The Phan tool tries to do this, and does so quite well, but at a cost (compl... [04:24:58] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #467: 04FAILURE in 28 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/467/ [04:31:46] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: Alert when l10update fails - https://phabricator.wikimedia.org/T171925#3480394 (10KartikMistry) [04:32:22] 10Scap, 10MediaWiki-extensions-LocalisationUpdate: l10nupdate failing with "git pull of extensions failed" since July 19th - https://phabricator.wikimedia.org/T171711#3473889 (10KartikMistry) @greg I created T171925 for alert. [06:38:47] 10Scap, 10Global-Collaboration, 10MediaWiki-extensions-LocalisationUpdate: Alert when l10update fails - https://phabricator.wikimedia.org/T171925#3480497 (10greg) >>! In T171711#3475616, @greg wrote: > https://www.mediawiki.org/wiki/Developers/Maintainers ctrl+f "localisationupdate" -> "Contributors > Global... [06:54:39] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:34:42] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [08:14:54] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [08:19:28] (03PS5) 10Hashar: Add composer-install & use in composer-test-mwextension [integration/config] - 10https://gerrit.wikimedia.org/r/354522 (https://phabricator.wikimedia.org/T165316) (owner: 10Aude) [08:39:07] 10Gerrit, 10MediaWiki-Interwiki, 10MediaWiki-Page-editing: Gerrit interwiki is broken - https://phabricator.wikimedia.org/T171936#3480713 (10Dvorapa) [08:47:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:58:20] 10Continuous-Integration-Infrastructure: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3480744 (10hashar) As exposed above, npm 3 resolves dependencies slightly differently than npm 2. Namely all the warnings about peer dependencies would cause the repo to fail. The differ... [09:06:51] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3480748 (10Addshore) [09:07:19] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3480751 (10Addshore) [09:08:49] (03CR) 10Addshore: [C: 031] "Can we try to move forward on this as this is now blocking a backport to 1.30-wmf.10?" [integration/config] - 10https://gerrit.wikimedia.org/r/354522 (https://phabricator.wikimedia.org/T165316) (owner: 10Aude) [09:08:56] hashar: ^^ [09:22:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:36:56] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3262820 (10hashar) For MediaWiki, we have dismissed using `composer install` in fa... [09:44:27] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:45:42] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [09:47:19] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core - https://phabricator.wikimedia.org/T171866#3480812 (10Volker_E) @hashar stylelint is intended to work flawlessly with CSS files (and in our configurations with Le... [10:15:09] 10Continuous-Integration-Infrastructure, 10Composer, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), 10Patch-For-Review: Build: Handle extensions autoloading entry point from composer.json - https://phabricator.wikimedia.org/T168738#3480835 (10Umherirrender) 05Open>03Resolved a:03Umhe... [10:20:42] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:25] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [10:34:11] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3480861 (10zeljkofilipin) @Jrbranaa I will be on vacation before Wikimania, and I guess @Rfarrand will be busy during Wikimania, so after Wikimania? [10:49:05] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Browser test jobs should use xUnit publisher instead of Junit - https://phabricator.wikimedia.org/T94684#3480895 (10zeljkofilipin) >>! In T94684#3476848, @Jdlrobson wrote: > How can I s... [11:13:47] 10MediaWiki-Codesniffer: Check @covers against autoloader with codesniffer - https://phabricator.wikimedia.org/T171899#3480927 (10Umherirrender) Thanks for the information, than phpcs only can look for wrong tags like "@cover"[1], but not for wrong classes. [1]https://phabricator.wikimedia.org/rMWda6235ce3a426e... [11:14:04] 10MediaWiki-Codesniffer: Check @covers against autoloader with codesniffer - https://phabricator.wikimedia.org/T171899#3480928 (10Umherirrender) p:05Triage>03Lowest [11:18:23] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:20:16] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:21:14] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:25:44] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude, 10Wikidata-Sprint: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3480940 (10daniel) [12:09:40] (03PS1) 10Zfilipin: Run mediawiki-core-qunit-selenium-jessie for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/368410 (https://phabricator.wikimedia.org/T171848) [12:17:09] (03CR) 10Zfilipin: [C: 032] Run mediawiki-core-qunit-selenium-jessie for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/368410 (https://phabricator.wikimedia.org/T171848) (owner: 10Zfilipin) [12:18:22] (03Merged) 10jenkins-bot: Run mediawiki-core-qunit-selenium-jessie for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/368410 (https://phabricator.wikimedia.org/T171848) (owner: 10Zfilipin) [12:19:43] !log Reloading Zuul to deploy 47a07e06e5e348902a3cdc51cff9c2bed87ab6df [12:19:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:32:18] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude, 10Wikidata-Sprint: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3481081 (10aude) @hashar yes, composer install when there is... [12:41:00] 10Gerrit, 10MediaWiki-Interwiki, 10MediaWiki-Page-editing: Gerrit interwiki is broken - https://phabricator.wikimedia.org/T171936#3480713 (10matmarex) To elaborate, with examples: * This works: `[[gerrit:367834]]` * This doesn't: `[[gerrit:#/c/367834/]]` In contrast, linking to fragments works for e.g. Phabr... [12:46:43] Hello, I'm interested in contributing to wikimedia. My background is computer science and my skills include python,linux,jenkins,git and knowledge on puppet. Can anybody help me in getting started. Thanks. [12:48:30] hi mkrish and welcome! [12:49:15] mkrish: have you already taken time to read through https://www.mediawiki.org/wiki/How_to_contribute [12:51:03] Yes,I did have a brief look into it. I'm interest in CI so found this would be the appropriate place to ask for help . [12:51:17] interested* [12:57:53] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude, 10Wikidata-Sprint: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3481127 (10hashar) [13:09:00] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Notifications, and 4 others: Port Echo Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T171848#3481185 (10zeljkofilipin) Green! https://integration.wikimedia.org/ci/job/mediawiki-core... [13:09:52] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Collaboration-Team-Triage, 10Notifications, and 4 others: Port Echo Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T171848#3481188 (10zeljkofilipin) a:05zeljkofilipin>03Etonkovidova [13:13:41] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude, 10Wikidata-Sprint: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3481194 (10hashar) Both jobs have been made to cd to the exte... [13:15:14] (03PS6) 10Hashar: Add composer-install & use in composer-test-mwextension [integration/config] - 10https://gerrit.wikimedia.org/r/354522 (https://phabricator.wikimedia.org/T165316) (owner: 10Aude) [13:16:42] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:26:00] (03CR) 10Hashar: [C: 032] "Jobs deployed and that solves the Wikidata use case at https://gerrit.wikimedia.org/r/#/c/368312/" [integration/config] - 10https://gerrit.wikimedia.org/r/354522 (https://phabricator.wikimedia.org/T165316) (owner: 10Aude) [13:27:05] 10Continuous-Integration-Config, 10Wikidata, 10Patch-For-Review, 10User-aude, 10Wikidata-Sprint: Wikidata build jenkins failure: Command "test" is not defined and don't run composer update in the job - https://phabricator.wikimedia.org/T165316#3481203 (10hashar) 05Open>03Resolved a:03aude I have de... [13:28:01] (03Merged) 10jenkins-bot: Add composer-install & use in composer-test-mwextension [integration/config] - 10https://gerrit.wikimedia.org/r/354522 (https://phabricator.wikimedia.org/T165316) (owner: 10Aude) [13:39:23] (03PS1) 10Hashar: Run npm on mediawiki/core on stylelint/css changes [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) [13:40:16] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Patch-For-Review, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core - https://phabricator.wikimedia.org/T171866#3481248 (10hashar) https://gerrit.wikimedia.org/r/#/c/368421/ will make CI to run the `npm test`... [13:40:30] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Patch-For-Review, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core - https://phabricator.wikimedia.org/T171866#3481249 (10hashar) p:05Triage>03Normal [13:46:01] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Reading-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3481259 (10zeljkofilipin) Modifying selenium-RelatedArticles job will not work since... [13:48:49] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #474: 04FAILURE in 4 min 47 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/474/ [13:51:40] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [13:56:11] Yippee, build fixed! [13:56:11] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #468: 09FIXED in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/468/ [15:01:16] 10Release-Engineering-Team (Kanban), 10RelatedArticles, 10Reading-Web-Backlog (Tracking), 10User-zeljkofilipin: Create Jenkins job that runs RelatedArticles Selenium tests daily - https://phabricator.wikimedia.org/T171847#3481431 (10zeljkofilipin) Testing job: [[ https://integration.wikimedia.org/ci/job/se... [16:05:55] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Patch-For-Review, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core - https://phabricator.wikimedia.org/T171866#3478741 (10Jdforrester-WMF) How much of an optimisation is the restriction? I'd think the majorit... [16:10:44] (03CR) 10Jdlrobson: [C: 031] Run npm on mediawiki/core on stylelint/css changes [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) (owner: 10Hashar) [16:11:26] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Jenkins, 10Patch-For-Review, 10Reading-Web-Backlog (Tracking): Jenkins should run `npm test` on core when only CSS files are touched - https://phabricator.wikimedia.org/T171866#3481633 (10Jdlrobson) [16:45:10] (03PS3) 10Umherirrender: Run mwgate-composer along with extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/364505 [16:45:46] (03PS4) 10Umherirrender: Run mwgate-composer along with extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/364505 [16:46:43] (03CR) 10jerkins-bot: [V: 04-1] Run mwgate-composer along with extension-gate [integration/config] - 10https://gerrit.wikimedia.org/r/364505 (owner: 10Umherirrender) [17:32:01] 10Release-Engineering-Team (Kanban), 10Reading-Web-Backlog, 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-08-01_(1.30.0-wmf.12)), and 2 others: Rewrite Related pages browser tests in Node.js - https://phabricator.wikimedia.org/T164024#3481978 (10Jdlrobson) [17:42:41] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:45:25] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:46:49] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3482012 (10Jdforrester-WMF) Draft: ># CI npm version upgraded from 2.x to 3.x >Please note that we've upgraded the version of npm used in the Wikimedia continuous in... [17:48:40] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3482015 (10Paladox) We should upgrade to npm 5 :). [18:04:44] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #33: 04FAILURE in 31 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/33/ [18:20:26] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [18:22:40] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [18:23:43] (03PS1) 10Subramanya Sastry: Add new config + new postprocessing files for comparing Tidy vs Remex [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/368468 [18:24:22] (03CR) 10Subramanya Sastry: [V: 032 C: 032] "self-merging (i meant to direct push, not git review :-))." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/368468 (owner: 10Subramanya Sastry) [18:43:15] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3482172 (10Rfarrand) After Wikimania works for me, let me know a time range that works for you and we can figure something out. If you want to involve... [19:08:55] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #34: 04FAILURE in 31 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/34/ [20:06:15] 10Deployment-Systems, 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase1), 10scap2, and 3 others: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) - https://phabricator.wikimedia.org/T129148#3482379 (10thcipriani) >>! In T129148#3482310, @gerritbot wrote: > Change 368476 had a re... [20:42:36] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:43:42] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:44:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Analytics-Kanban, 10Analytics-Wikistats: Fix Wikistats build in Jenkins - https://phabricator.wikimedia.org/T171599#3482606 (10hashar) Indeed we are still using npm 2 and havent bothered yet to upgrade to 3 or later. That is the subject... [20:45:49] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3482629 (10Paladox) Also +1 to your mock up email :) [20:53:42] (03PS1) 10XZise: [IMPROV] Don't include BAD_FOOTERS in FOOTERS [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/368515 [20:55:15] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jobs with Node 6 should also have npm 3 - https://phabricator.wikimedia.org/T161861#3482707 (10hashar) @Jdforrester-WMF thank you for the mail draft!!! It looks good to me. npm 4 has a few breaking changes but nothing that should impact us. Though... [21:11:31] !log Dropped table wikigrok_questions from beta enwiki [21:11:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:18:40] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [21:22:36] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:35:52] (03CR) 10Krinkle: [C: 031] Run npm on mediawiki/core on stylelint/css changes [integration/config] - 10https://gerrit.wikimedia.org/r/368421 (https://phabricator.wikimedia.org/T171866) (owner: 10Hashar) [21:54:54] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:34:53] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [23:01:31] 10Release-Engineering-Team, 10Operations, 10vm-requests, 10Security-General: New ganeti VM for MW release pipeline work - https://phabricator.wikimedia.org/T163743#3483047 (10Dzahn) [23:01:33] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3483046 (10Dzahn) 05Open>03Resolved [23:02:58] 10Release-Engineering-Team (Watching / External), 10Operations: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#3483092 (10Dzahn) [23:03:01] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3218909 (10Dzahn) [23:03:04] 10Release-Engineering-Team (Kanban), 10Operations, 10vm-requests, 10Patch-For-Review: setup releases2001.codfw.wmnet - https://phabricator.wikimedia.org/T171917#3483090 (10Dzahn) 05Open>03Resolved [23:44:45] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#3483199 (10Dzahn) [23:45:17] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Provide cross-dc redundancy (active-active or active-passive) to all important misc services - https://phabricator.wikimedia.org/T156937#2990470 (10Dzahn)