[00:04:28] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Signing MW Core Tags - https://phabricator.wikimedia.org/T204995 (10Legoktm) The makerelease2.py script will not let you create signed tarballs unless the tag is also signed. [00:04:28] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.32-release: Signing MW Core Tags - https://phabricator.wikimedia.org/T204995 (10Legoktm) The makerelease2.py script will not let you create signed tarballs unless the tag is also signed. [02:07:26] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Mail, 10MediaWiki-Email, 10Regression: betalabs email is not working - https://phabricator.wikimedia.org/T212249 (10greg) Adding both #mail and #mediawiki-email as it's not clear where the cause is yet. [02:07:26] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Mail, 10MediaWiki-Email, 10Regression: betalabs email is not working - https://phabricator.wikimedia.org/T212249 (10greg) Adding both #mail and #mediawiki-email as it's not clear where the cause is yet. [02:08:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Discovery-Search, 10Elasticsearch: Set up data storage to collect loosely structured data from CI - https://phabricator.wikimedia.org/T211904 (10greg) [02:08:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Discovery-Search, 10Elasticsearch: Set up data storage to collect loosely structured data from CI - https://phabricator.wikimedia.org/T211904 (10greg) [02:14:37] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Delete test leftovers from CI - https://phabricator.wikimedia.org/T211343 (10greg) 05Open→03Invalid There is nothing persisting, as @hashar said. If it were, wouldn't PS3 result in 15 instead of 10? It is, however, showing 1... [02:14:37] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Delete test leftovers from CI - https://phabricator.wikimedia.org/T211343 (10greg) 05Open→03Invalid There is nothing persisting, as @hashar said. If it were, wouldn't PS3 result in 15 instead of 10? It is, however, showing 1... [02:16:14] (03PS1) 10Dduvall: Use blubberoid.discovery.wmnet to process blubber.yaml [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) [02:16:14] (03PS1) 10Dduvall: Use blubberoid.discovery.wmnet to process blubber.yaml [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) [02:23:28] marxarelli: nice :) ^ [02:23:28] marxarelli: nice :) ^ [02:25:00] greg-g: yeah, hope it works :) [02:25:00] greg-g: yeah, hope it works :) [02:25:40] i've been looking for a good way to do integration testing against shared pipeline libraries, but not finding anything worth doing atm [02:25:40] i've been looking for a good way to do integration testing against shared pipeline libraries, but not finding anything worth doing atm [02:26:21] maybe this but we'd probably need to build our own base image for it in integration/config/dockerfiles https://github.com/jenkinsci/jenkinsfile-runner [02:26:21] maybe this but we'd probably need to build our own base image for it in integration/config/dockerfiles https://github.com/jenkinsci/jenkinsfile-runner [02:29:15] ah, interesting [02:29:15] ah, interesting [04:13:46] going to cherry-pick https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/480419/ on beta [04:13:46] going to cherry-pick https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/480419/ on beta [04:20:05] Project beta-update-databases-eqiad build #30587: 04FAILURE in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30587/ [04:20:05] Project beta-update-databases-eqiad build #30587: 04FAILURE in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30587/ [04:22:56] Project beta-scap-eqiad build #231392: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231392/ [04:22:56] Project beta-scap-eqiad build #231392: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231392/ [04:23:58] Project beta-scap-eqiad build #231393: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231393/ [04:23:58] Project beta-scap-eqiad build #231393: 04STILL FAILING in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231393/ [04:25:01] Project beta-scap-eqiad build #231394: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231394/ [04:25:01] Project beta-scap-eqiad build #231394: 04STILL FAILING in 1.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231394/ [04:34:05] Project beta-scap-eqiad build #231395: 04STILL FAILING in 4.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231395/ [04:34:05] Project beta-scap-eqiad build #231395: 04STILL FAILING in 4.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231395/ [04:44:01] Project beta-scap-eqiad build #231396: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231396/ [04:44:01] Project beta-scap-eqiad build #231396: 04STILL FAILING in 1.3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231396/ [04:54:09] Project beta-scap-eqiad build #231397: 04STILL FAILING in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231397/ [04:54:09] Project beta-scap-eqiad build #231397: 04STILL FAILING in 1.1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231397/ [04:58:20] found it [04:58:20] found it [05:07:43] legoktm: it was indeed a beta bug: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480695/ [05:07:43] legoktm: it was indeed a beta bug: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480695/ [05:16:08] Yippee, build fixed! [05:16:08] Yippee, build fixed! [05:16:08] Project beta-scap-eqiad build #231398: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231398/ [05:16:08] Project beta-scap-eqiad build #231398: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/231398/ [05:21:22] Yippee, build fixed! [05:21:22] Yippee, build fixed! [05:21:23] Project beta-update-databases-eqiad build #30588: 09FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30588/ [05:21:23] Project beta-update-databases-eqiad build #30588: 09FIXED in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/30588/ [06:42:34] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) Looks like it is indeed not in use: ` root@db1073:/srv/sqldata/nodepooldb... [06:42:34] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) Looks like it is indeed not in use: ` root@db1073:/srv/sqldata/nodepooldb... [06:42:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) a:03Marostegui [06:42:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) a:03Marostegui [06:42:55] TimStarling: wow, nice find [06:42:55] TimStarling: wow, nice find [06:45:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) User removed ` root@db1073.eqiad.wmnet[(none)]> drop user if exists 'node... [06:45:18] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) User removed ` root@db1073.eqiad.wmnet[(none)]> drop user if exists 'node... [07:38:39] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10Marostegui) [07:38:39] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10Marostegui) [07:38:42] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) 05Open→03Resolved Database dropped: ` root@db1073.eqiad.wmnet[(none)... [07:38:42] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DBA, 10Patch-For-Review: [DBA] remove nodepooldb on production-m5 and nodepool user - https://phabricator.wikimedia.org/T212230 (10Marostegui) 05Open→03Resolved Database dropped: ` root@db1073.eqiad.wmnet[(none)... [08:10:40] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10Osnard) [08:10:40] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10Osnard) [08:40:36] 10Release-Engineering-Team, 10Scap, 10Operations, 10User-ArielGlenn: Make scap and opcache work consistently together - https://phabricator.wikimedia.org/T211964 (10Joe) [08:40:36] 10Release-Engineering-Team, 10Scap, 10Operations, 10User-ArielGlenn: Make scap and opcache work consistently together - https://phabricator.wikimedia.org/T211964 (10Joe) [09:39:40] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Delete test leftovers from CI - https://phabricator.wikimedia.org/T211343 (10Daimona) >>! In T211343#4833243, @greg wrote: > If it were, wouldn't PS3 result in 15 instead of 10? True. I retried and still got the same result. Th... [09:39:40] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Delete test leftovers from CI - https://phabricator.wikimedia.org/T211343 (10Daimona) >>! In T211343#4833243, @greg wrote: > If it were, wouldn't PS3 result in 15 instead of 10? True. I retried and still got the same result. Th... [09:49:35] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [09:49:35] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [09:56:23] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) With [[ https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480573/ | Gerrit #480573 ]] doc1001.eqiad.wmn... [09:56:23] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) With [[ https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480573/ | Gerrit #480573 ]] doc1001.eqiad.wmn... [10:00:45] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [10:00:45] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10cloud-services-team (Kanban): Phase out Nodepool from production - https://phabricator.wikimedia.org/T209361 (10hashar) [10:23:55] 10Continuous-Integration-Config, 10External Guidance, 10User-MarcoAurelio: Add CI support to ExternalGuidance - https://phabricator.wikimedia.org/T212286 (10MarcoAurelio) [10:23:55] 10Continuous-Integration-Config, 10External Guidance, 10User-MarcoAurelio: Add CI support to ExternalGuidance - https://phabricator.wikimedia.org/T212286 (10MarcoAurelio) [10:26:29] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10hashar) The MediaWiki instance has debugging/error logging enabled and the log files are captured by the build. On https://integration.wikimedia.org/ci/job/quibble-c... [10:26:29] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10hashar) The MediaWiki instance has debugging/error logging enabled and the log files are captured by the build. On https://integration.wikimedia.org/ci/job/quibble-c... [10:26:51] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10hashar) [10:26:51] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10hashar) [10:27:06] 10Project-Admins, 10External Guidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10MarcoAurelio) [10:27:06] 10Project-Admins, 10External Guidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10MarcoAurelio) [10:28:07] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10hashar) 05Open→03Invalid The previous change is https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/BlueSpiceCustomMenu/+/480483/ and PS2 passed all tests s... [10:28:07] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10hashar) 05Open→03Invalid The previous change is https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/BlueSpiceCustomMenu/+/480483/ and PS2 passed all tests s... [10:35:06] 10Project-Admins, 10External Guidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10santhosh) a:03Arrbee [10:35:06] 10Project-Admins, 10External Guidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10santhosh) a:03Arrbee [10:43:14] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10hashar) the `wmnet.` top level domain is for production / internal services. WMCS is considered an alien and is not allowed to reach it. The s... [10:43:14] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10hashar) the `wmnet.` top level domain is for production / internal services. WMCS is considered an alien and is not allowed to reach it. The s... [10:58:32] 10Project-Admins, 10ExternalGuidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10Pginer-WMF) 05Open→03Resolved Renamed to `#ExternalGuidance` [10:58:32] 10Project-Admins, 10ExternalGuidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10Pginer-WMF) 05Open→03Resolved Renamed to `#ExternalGuidance` [10:58:38] (03PS1) 10MarcoAurelio: Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) [10:58:38] (03PS1) 10MarcoAurelio: Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) [10:59:52] 10Project-Admins, 10ExternalGuidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10MarcoAurelio) ¡Gracias! [10:59:52] 10Project-Admins, 10ExternalGuidance: #External_Guidance should be renamed to #mediawiki-extensions-ExternalGuidance or #ExternalGuidance - https://phabricator.wikimedia.org/T212289 (10MarcoAurelio) ¡Gracias! [11:00:20] (03CR) 10jerkins-bot: [V: 04-1] Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [11:00:20] (03CR) 10jerkins-bot: [V: 04-1] Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [11:01:43] I don't understand ^^ failures [11:01:43] I don't understand ^^ failures [11:01:50] looks like some jjb issues? [11:01:50] looks like some jjb issues? [11:28:56] (03PS1) 10Volans: operations/spicerack: publish documentation [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) [11:28:56] (03PS1) 10Volans: operations/spicerack: publish documentation [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) [11:29:58] (03CR) 10Volans: "The documentation generation is being added in I3cff46533980c016565a02876459511869ba88e0" [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [11:29:58] (03CR) 10Volans: "The documentation generation is being added in I3cff46533980c016565a02876459511869ba88e0" [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [11:31:52] (03CR) 10Tulsi Bhagat: [C: 03+1] "Thank you! Framawiki ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [11:31:52] (03CR) 10Tulsi Bhagat: [C: 03+1] "Thank you! Framawiki ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [11:34:41] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) I guess it depends on the use cases. As SREs I don't think we would like to expose a public service that is not going to receive a... [11:34:41] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) I guess it depends on the use cases. As SREs I don't think we would like to expose a public service that is not going to receive a... [11:57:31] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "I see some other entries in this file already do the same, so this is fine. But I still find this a weird to do. Shouldn't all these email" [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [11:57:31] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "I see some other entries in this file already do the same, so this is fine. But I still find this a weird to do. Shouldn't all these email" [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [12:15:50] (03CR) 10MarcoAurelio: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [12:15:50] (03CR) 10MarcoAurelio: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [12:21:38] (03CR) 10jerkins-bot: [V: 04-1] Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [12:21:38] (03CR) 10jerkins-bot: [V: 04-1] Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [12:33:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10Mholloway) [12:33:34] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10Mholloway) [12:35:10] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, 10User-jijiki: Requesting access to deployment for Christoph Jauera (WMDE-Fisch) - https://phabricator.wikimedia.org/T211014 (10WMDE-Fisch) >>! In T211014#4812240, @Dzahn wrote: > @WMDE-Fisch Your a... [12:35:10] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review, 10User-jijiki: Requesting access to deployment for Christoph Jauera (WMDE-Fisch) - https://phabricator.wikimedia.org/T211014 (10WMDE-Fisch) >>! In T211014#4812240, @Dzahn wrote: > @WMDE-Fisch Your a... [12:43:50] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10Mholloway) [12:43:50] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10Mholloway) [14:00:49] be [14:00:49] be [14:34:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10zeljkofilipin) I've noticed this in [[ https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors | mediawiki-errors ]]: ` [... [14:34:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10zeljkofilipin) I've noticed this in [[ https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors | mediawiki-errors ]]: ` [... [14:47:53] hashar: if you have a minute, I have a CR for you :) But I'm not totally sure that I put all the correct bits there [14:47:53] hashar: if you have a minute, I have a CR for you :) But I'm not totally sure that I put all the correct bits there [14:47:57] https://gerrit.wikimedia.org/r/c/integration/config/+/480725 [14:47:57] https://gerrit.wikimedia.org/r/c/integration/config/+/480725 [14:49:33] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DynamicPageList (Wikimedia), 10MW-1.32-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), 10Patch-For-Review: [intersection] parser tests fail and CI ignore tests - https://phabricator.wikimedia.org/T200063 (10hashar)... [14:49:33] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10DynamicPageList (Wikimedia), 10MW-1.32-notes (WMF-deploy-2018-07-24 (1.32.0-wmf.14)), 10Patch-For-Review: [intersection] parser tests fail and CI ignore tests - https://phabricator.wikimedia.org/T200063 (10hashar)... [14:54:47] (03CR) 10Lars Wirzenius: [C: 03+1] "I've made some suggestions, but there's nothing major, and if you if you disagree, that's OK." (034 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) (owner: 10Dduvall) [14:54:47] (03CR) 10Lars Wirzenius: [C: 03+1] "I've made some suggestions, but there's nothing major, and if you if you disagree, that's OK." (034 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/480689 (https://phabricator.wikimedia.org/T212247) (owner: 10Dduvall) [14:55:34] (03PS2) 10Hashar: Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [14:55:34] (03PS2) 10Hashar: Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [14:55:56] (03CR) 10Hashar: [C: 03+2] "Missed a space: -name -> - name" [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [14:55:56] (03CR) 10Hashar: [C: 03+2] "Missed a space: -name -> - name" [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [14:57:23] (03Merged) 10jenkins-bot: Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [14:57:23] (03Merged) 10jenkins-bot: Register jobs for extension-ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480723 (https://phabricator.wikimedia.org/T212286) (owner: 10MarcoAurelio) [15:04:20] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10hashar) >>! In T212251#4833906, @akosiaris wrote: > I guess it depends on the use cases. > > As SREs I don't think we would like to expose a... [15:04:20] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10hashar) >>! In T212251#4833906, @akosiaris wrote: > I guess it depends on the use cases. > > As SREs I don't think we would like to expose a... [15:05:33] 10Continuous-Integration-Config, 10ExternalGuidance, 10Patch-For-Review, 10User-MarcoAurelio: Add CI support to ExternalGuidance - https://phabricator.wikimedia.org/T212286 (10hashar) Dummy change to validate the CI jobs: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ExternalGuidance/+/480755 [15:05:33] 10Continuous-Integration-Config, 10ExternalGuidance, 10Patch-For-Review, 10User-MarcoAurelio: Add CI support to ExternalGuidance - https://phabricator.wikimedia.org/T212286 (10hashar) Dummy change to validate the CI jobs: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ExternalGuidance/+/480755 [15:06:42] 10Gerrit, 10Proton, 10Core Platform Team Backlog (Attic), 10Readers-Web-Backlog (Tracking), and 2 others: Rename chromium-render gerrit project to Proton - https://phabricator.wikimedia.org/T211375 (10hashar) Should the task just be declined based on @mobrovac comment? Not sure whether it is worth all the... [15:06:42] 10Gerrit, 10Proton, 10Core Platform Team Backlog (Attic), 10Readers-Web-Backlog (Tracking), and 2 others: Rename chromium-render gerrit project to Proton - https://phabricator.wikimedia.org/T211375 (10hashar) Should the task just be declined based on @mobrovac comment? Not sure whether it is worth all the... [15:11:36] (03CR) 10Hashar: [C: 03+2] "I can not tell how Gerrit handles case of emails. One sure thing it is currently registered with an upper case T so I guess that is why it" [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [15:11:36] (03CR) 10Hashar: [C: 03+2] "I can not tell how Gerrit handles case of emails. One sure thing it is currently registered with an upper case T so I guess that is why it" [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [15:12:29] volans: looking :) [15:12:29] volans: looking :) [15:12:36] thanks! [15:12:36] thanks! [15:13:01] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) >>! In T212251#4834325, @hashar wrote: >>>! In T212251#4833906, @akosiaris wrote: >> I guess it depends on the use cases. >> >> As... [15:13:01] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10akosiaris) >>! In T212251#4834325, @hashar wrote: >>>! In T212251#4833906, @akosiaris wrote: >> I guess it depends on the use cases. >> >> As... [15:13:21] (03Merged) 10jenkins-bot: Fix whitelisting entry of tulsibhagat50 [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [15:13:21] (03Merged) 10jenkins-bot: Fix whitelisting entry of tulsibhagat50 [integration/config] - 10https://gerrit.wikimedia.org/r/479979 (owner: 10Framawiki) [15:13:23] volans: that looks legit :) [15:13:23] volans: that looks legit :) [15:14:29] hashar: I mostly copied the bits from cumin, but cumin has a custom job, so I wasn't sure of all the pieces :) [15:14:29] hashar: I mostly copied the bits from cumin, but cumin has a custom job, so I wasn't sure of all the pieces :) [15:14:30] (03CR) 10Hashar: [C: 03+2] "Looks all legit thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:14:30] (03CR) 10Hashar: [C: 03+2] "Looks all legit thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:14:40] volans: yeah that is how it goes really [15:14:40] volans: yeah that is how it goes really [15:14:43] also maybe cumin doesn't need that custom job anymore? nothing for today [15:14:43] also maybe cumin doesn't need that custom job anymore? nothing for today [15:14:47] same use case, same solution! [15:14:47] same use case, same solution! [15:15:35] for testing it I have https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/480757 that is ready to be merged as soon as CI finishes there [15:15:35] for testing it I have https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/480757 that is ready to be merged as soon as CI finishes there [15:17:14] (03Merged) 10jenkins-bot: operations/spicerack: publish documentation [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:17:14] (03Merged) 10jenkins-bot: operations/spicerack: publish documentation [integration/config] - 10https://gerrit.wikimedia.org/r/480725 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:19:30] hashar: <3! Let me know when I should merge my pending patch so to check if this works as expected [15:19:30] hashar: <3! Let me know when I should merge my pending patch so to check if this works as expected [15:20:35] !log Testing spicerack postmerge job for T205894: contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/spicerack --change 480724,2 [15:20:35] !log Testing spicerack postmerge job for T205894: contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/spicerack --change 480724,2 [15:20:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:20:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:20:38] T205894: Generate documentation for Spicerack - https://phabricator.wikimedia.org/T205894 [15:20:38] T205894: Generate documentation for Spicerack - https://phabricator.wikimedia.org/T205894 [15:21:53] volans: well you have aced it :) https://doc.wikimedia.org/spicerack/master/ [15:21:53] volans: well you have aced it :) https://doc.wikimedia.org/spicerack/master/ [15:23:05] great!!! thanks a lot! I'll make a release now so I can check that also the tag one works as a by-product :) [15:23:05] great!!! thanks a lot! I'll make a release now so I can check that also the tag one works as a by-product :) [15:25:18] volans: note : I am not sure the tag based job notify anything [15:25:18] volans: note : I am not sure the tag based job notify anything [15:25:21] notifies [15:25:21] notifies [15:26:35] ah, IIRC in cumin it works the only weird thing is the version showed [15:26:35] ah, IIRC in cumin it works the only weird thing is the version showed [15:26:37] https://doc.wikimedia.org/cumin/v3.0.2/ [15:26:37] https://doc.wikimedia.org/cumin/v3.0.2/ [15:27:09] that is not the tag one, but you and I didn't had time to go down that rabbit hole at the time [15:27:09] that is not the tag one, but you and I didn't had time to go down that rabbit hole at the time [15:32:16] (03PS1) 10Volans: Add link to Spicerack documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) [15:32:16] (03PS1) 10Volans: Add link to Spicerack documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) [15:32:21] hashar: icing on the cake :) ^^^ [15:32:21] hashar: icing on the cake :) ^^^ [15:42:40] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10Osnard) @hashar , Thanks a lot! [15:42:40] 10Continuous-Integration-Infrastructure, 10Gerrit: BlueSpice* repos: Selenium tests break randomly - https://phabricator.wikimedia.org/T212278 (10Osnard) @hashar , Thanks a lot! [15:47:09] (03CR) 10Hashar: [C: 03+2] "I keep forgetting about the docroot :( thank you!" [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:47:09] (03CR) 10Hashar: [C: 03+2] "I keep forgetting about the docroot :( thank you!" [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:47:47] (03Merged) 10jenkins-bot: Add link to Spicerack documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:47:47] (03Merged) 10jenkins-bot: Add link to Spicerack documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:47:56] (03CR) 10jenkins-bot: Add link to Spicerack documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:47:56] (03CR) 10jenkins-bot: Add link to Spicerack documentation [integration/docroot] - 10https://gerrit.wikimedia.org/r/480763 (https://phabricator.wikimedia.org/T205894) (owner: 10Volans) [15:50:46] /etc/puppet-enc.yaml:host: labs-puppetmaster.wikimedia.org [15:50:46] /etc/puppet-enc.yaml:host: labs-puppetmaster.wikimedia.org [15:50:53] /etc/puppet/hiera.yaml: :url_prefix: http://labs-puppetmaster.wikimedia.org:8100/v1/ [15:50:53] /etc/puppet/hiera.yaml: :url_prefix: http://labs-puppetmaster.wikimedia.org:8100/v1/ [15:51:01] on a the CI puppetmaster bah [15:51:01] on a the CI puppetmaster bah [16:40:57] 10Gerrit, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Backlog (Attic), and 2 others: Rename chromium-render gerrit project to Proton - https://phabricator.wikimedia.org/T211375 (10LGoto) [16:40:57] 10Gerrit, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Backlog (Attic), and 2 others: Rename chromium-render gerrit project to Proton - https://phabricator.wikimedia.org/T211375 (10LGoto) [16:44:02] 10Gerrit, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Backlog (Attic), and 2 others: Rename chromium-render gerrit project to Proton - https://phabricator.wikimedia.org/T211375 (10Jhernandez) 05Open→03Declined Declined for now, given the comments. We will revisit later [16:44:02] 10Gerrit, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Backlog (Attic), and 2 others: Rename chromium-render gerrit project to Proton - https://phabricator.wikimedia.org/T211375 (10Jhernandez) 05Open→03Declined Declined for now, given the comments. We will revisit later [16:48:12] 10Continuous-Integration-Config, 10Mobile-Content-Service, 10Reading-Infrastructure-Team-Backlog, 10JavaScript: Upgrade the mobileapps CI job from npm3 to npm6 - https://phabricator.wikimedia.org/T211785 (10bearND) Assuming this is CI doing this, hence moving to tracking. Feel free to move to our backlog i... [16:48:12] 10Continuous-Integration-Config, 10Mobile-Content-Service, 10Reading-Infrastructure-Team-Backlog, 10JavaScript: Upgrade the mobileapps CI job from npm3 to npm6 - https://phabricator.wikimedia.org/T211785 (10bearND) Assuming this is CI doing this, hence moving to tracking. Feel free to move to our backlog i... [16:51:04] 10Release-Engineering-Team (Kanban), 10Education-Program-Dashboard, 10MediaWiki-extensions-EducationProgram, 10Epic, 10User-greg: Deprecate and remove the EducationProgram extension from Wikimedia servers after June 30, 2018 - https://phabricator.wikimedia.org/T125618 (10Reedy) [16:51:04] 10Release-Engineering-Team (Kanban), 10Education-Program-Dashboard, 10MediaWiki-extensions-EducationProgram, 10Epic, 10User-greg: Deprecate and remove the EducationProgram extension from Wikimedia servers after June 30, 2018 - https://phabricator.wikimedia.org/T125618 (10Reedy) [16:59:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) [16:59:44] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) [17:07:06] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) I will take care of deploying to group1 during the American window at 20:00 UTC (21:00 CET - 11am PST). [17:07:06] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) I will take care of deploying to group1 during the American window at 20:00 UTC (21:00 CET - 11am PST). [17:09:14] mutante: I am tweaking the CI config for doc.wikimedia.org :) [17:09:14] mutante: I am tweaking the CI config for doc.wikimedia.org :) [17:09:38] mutante: and it seems to be working fine. I will have to look at the http config and whether it takes in account the various bits we had [17:09:38] mutante: and it seems to be working fine. I will have to look at the http config and whether it takes in account the various bits we had [17:09:49] namely support the https header if still relevant [17:09:49] namely support the https header if still relevant [17:10:05] and support for the old doc.mediawiki.org (though I am not sure whether we still have hits on that old one) [17:10:05] and support for the old doc.mediawiki.org (though I am not sure whether we still have hits on that old one) [17:11:39] hashar: ok! i merged the cluster/notification group change. for the other one it needs a subticket but i'll create that [17:11:39] hashar: ok! i merged the cluster/notification group change. for the other one it needs a subticket but i'll create that [17:11:58] is it really necessary to manually edit files? [17:11:58] is it really necessary to manually edit files? [17:13:01] (03PS2) 10Hashar: Overhaul publishing [integration/config] - 10https://gerrit.wikimedia.org/r/480657 (https://phabricator.wikimedia.org/T211974) [17:13:01] (03PS2) 10Hashar: Overhaul publishing [integration/config] - 10https://gerrit.wikimedia.org/r/480657 (https://phabricator.wikimedia.org/T211974) [17:14:06] adding the existing groups to new host i could merge but the new sudo privs is a bit different [17:14:06] adding the existing groups to new host i could merge but the new sudo privs is a bit different [17:14:35] we are not changing the practical permissions but have to explain that.. new user and all [17:14:35] we are not changing the practical permissions but have to explain that.. new user and all [17:15:07] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) The VM is working and the basic service is there ( rsyncd ). I will complete the service implementation via... [17:15:07] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) The VM is working and the basic service is there ( rsyncd ). I will complete the service implementation via... [17:15:26] (03PS3) 10Hashar: (WIP) Overhaul publishing [integration/config] - 10https://gerrit.wikimedia.org/r/480657 (https://phabricator.wikimedia.org/T137890) [17:15:26] (03PS3) 10Hashar: (WIP) Overhaul publishing [integration/config] - 10https://gerrit.wikimedia.org/r/480657 (https://phabricator.wikimedia.org/T137890) [17:15:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Developer-Wishlist (2017), 10Patch-For-Review, and 2 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10hashar) a:03hashar [17:15:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Developer-Wishlist (2017), 10Patch-For-Review, and 2 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10hashar) a:03hashar [17:16:18] mutante: yeah no worry. The CI part will take a part to be tested corretly but at least I have a rsync server to "play" with [17:16:18] mutante: yeah no worry. The CI part will take a part to be tested corretly but at least I have a rsync server to "play" with [17:16:40] mutante: I guess the switch will occur sometime in january. I also need to test the http service [17:16:40] mutante: I guess the switch will occur sometime in january. I also need to test the http service [17:17:16] but port 80 is not recheable from contint1001 or bastion (which is fine). So I would need the shell access [17:17:16] but port 80 is not recheable from contint1001 or bastion (which is fine). So I would need the shell access [17:17:26] which I guess can only be granted at the next sre meeting :) [17:17:26] which I guess can only be granted at the next sre meeting :) [17:17:46] I have enough work to do on the ci conf anyway so it is not a big deal! [17:17:46] I have enough work to do on the ci conf anyway so it is not a big deal! [17:17:46] hashar: shell access can be granted now.. if we wouldn't add the sudo stuff too [17:17:46] hashar: shell access can be granted now.. if we wouldn't add the sudo stuff too [17:17:54] ahh [17:17:54] ahh [17:18:04] just putting contint-admins on the new VM seems no issue [17:18:04] just putting contint-admins on the new VM seems no issue [17:18:12] well I can split the change so [17:18:12] well I can split the change so [17:18:32] yea, that wouldn't be bad [17:18:32] yea, that wouldn't be bad [17:19:59] hashar: and for the sudo part.. i will just make an access request and explain how that is no practical difference.. how jenkins-slave is used before for the same files [17:19:59] hashar: and for the sudo part.. i will just make an access request and explain how that is no practical difference.. how jenkins-slave is used before for the same files [17:20:11] and the same people can run commands as _that_ user on the old hosts [17:20:11] and the same people can run commands as _that_ user on the old hosts [17:20:48] it just may not be obvious to the casual observer [17:20:48] it just may not be obvious to the casual observer [17:21:22] mutante: done https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480716/ :) [17:21:22] mutante: done https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480716/ :) [17:21:37] "may be useful" may sound a bit weak though [17:21:37] "may be useful" may sound a bit weak though [17:21:40] hashar: ok! cool, thx [17:21:40] hashar: ok! cool, thx [17:22:56] mutante: and the sudo rule addition is a new change : https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480798 [17:22:56] mutante: and the sudo rule addition is a new change : https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480798 [17:23:03] perfect [17:23:03] perfect [17:23:07] ;-))))))))))) [17:23:07] ;-))))))))))) [17:23:14] I should be more atomic [17:23:14] I should be more atomic [17:23:31] if I manage to get shell access, I can then verify the http config [17:23:31] if I manage to get shell access, I can then verify the http config [17:23:36] and ensure it is working properly [17:23:36] and ensure it is working properly [17:23:50] (there are a few edge cases iirc, would have to look at the current config and compare) [17:23:50] (there are a few edge cases iirc, would have to look at the current config and compare) [17:24:00] at lesat [17:24:00] at lesat [17:24:09] I got a job to successfully rsync material to the new host \o/ [17:24:09] I got a job to successfully rsync material to the new host \o/ [17:24:52] hashar: i used to use "apache-fast-test" on mwmaint or mwdebug to test but then we got more strict with ferm and limit the backends to only $CACHE [17:24:52] hashar: i used to use "apache-fast-test" on mwmaint or mwdebug to test but then we got more strict with ferm and limit the backends to only $CACHE [17:25:19] for that reason i once requested to add the test script on cache nodes.. or allow another host .. but i haven't done it / abandoned [17:25:19] for that reason i once requested to add the test script on cache nodes.. or allow another host .. but i haven't done it / abandoned [17:25:39] [doc1001:~] $ id hashar [17:25:39] [doc1001:~] $ id hashar [17:25:39] uid=1010(hashar) gid=500(wikidev) groups=500(wikidev),719(contint-admins) [17:25:39] uid=1010(hashar) gid=500(wikidev) groups=500(wikidev),719(contint-admins) [17:25:41] ^ there you gp [17:25:41] ^ there you gp [17:25:43] yeah apache-fast-test would be great [17:25:43] yeah apache-fast-test would be great [17:26:12] OH YEAH [17:26:12] OH YEAH [17:26:19] and I can read files that got published [17:26:19] and I can read files that got published [17:26:25] :) [17:26:25] :) [17:26:26] mutante: amazing! the sudo rule can wait [17:26:26] mutante: amazing! the sudo rule can wait [17:26:31] ok, cool [17:26:31] ok, cool [17:26:50] and I get http access over 127.0.0.1 [17:26:50] and I get http access over 127.0.0.1 [17:27:36] ack, "curl localhost" gets me the debian default page [17:27:36] ack, "curl localhost" gets me the debian default page [17:27:43] will test it later tonight with something like curl -H 'Host: doc.wikimedia.org' http://127.0.0.1/ [17:27:43] will test it later tonight with something like curl -H 'Host: doc.wikimedia.org' http://127.0.0.1/ [17:27:47] just the lack of apache-fest-test..hmm [17:27:47] just the lack of apache-fest-test..hmm [17:27:51] ok:) [17:27:51] ok:) [17:28:14] you may just get a "redirects to https" though [17:28:14] you may just get a "redirects to https" though [17:28:24] and then you cant test that because we dont speak https at backend [17:28:24] and then you cant test that because we dont speak https at backend [17:28:25] yeah that is what I need to check :) [17:28:25] yeah that is what I need to check :) [17:28:28] ok [17:28:28] ok [17:28:36] I cant remember the crazy config we had to put in place on contint1001 to handle https [17:28:36] I cant remember the crazy config we had to put in place on contint1001 to handle https [17:28:39] but I will find out [17:28:39] but I will find out [17:29:17] there must be some magic in ./modules/contint/templates/apache/doc.wikimedia.org.erb [17:29:17] there must be some magic in ./modules/contint/templates/apache/doc.wikimedia.org.erb [17:30:19] mutante: ah yeah apache does not have any configuration yet does it? [17:30:19] mutante: ah yeah apache does not have any configuration yet does it? [17:34:14] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480802 doc: add Apache config for doc.wikimedia.org [17:34:14] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480802 doc: add Apache config for doc.wikimedia.org [17:34:39] hashar: i think the magic you refer to is not a thing anymore [17:34:39] hashar: i think the magic you refer to is not a thing anymore [17:34:50] yea, the apache site config had not been done yet [17:34:50] yea, the apache site config had not been done yet [17:34:53] looks [17:34:53] looks [17:35:23] mutante: PS2 fix the test failure [17:35:23] mutante: PS2 fix the test failure [17:35:35] I have just reused the same template from the contint module [17:35:35] I have just reused the same template from the contint module [17:36:52] i was just looking at that and why PS1 failed. learned something about tests yesterday from your comments [17:36:52] i was just looking at that and why PS1 failed. learned something about tests yesterday from your comments [17:37:01] and actually did "bundle install" on laptop [17:37:01] and actually did "bundle install" on laptop [17:38:37] using template from other modules is not ideal but in this case.. yea.. why make a copy of the same thing [17:38:37] using template from other modules is not ideal but in this case.. yea.. why make a copy of the same thing [17:38:53] and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480804/ would be the cleanup change to merge later on [17:38:53] and https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480804/ would be the cleanup change to merge later on [17:39:01] I did it right now to be sure to not forget about it :))) [17:39:01] I did it right now to be sure to not forget about it :))) [17:39:03] once we flipped over we should move it to profile/templates/doc/ [17:39:03] once we flipped over we should move it to profile/templates/doc/ [17:39:09] yeah [17:39:09] yeah [17:39:10] :) [17:39:10] :) [17:39:15] and apparnetly it doe snot even need to be a template [17:39:15] and apparnetly it doe snot even need to be a template [17:39:39] guess we can try having the apache config setup, and I will test it tonight after the mediawiki train [17:39:39] guess we can try having the apache config setup, and I will test it tonight after the mediawiki train [17:41:50] yea, i often think "does not need to be a template" but then i also think "not yet but one thing pops up and we might want to convert it to template" or all the other sites are in one place etc.. no big deal either way :) [17:41:50] yea, i often think "does not need to be a template" but then i also think "not yet but one thing pops up and we might want to convert it to template" or all the other sites are in one place etc.. no big deal either way :) [17:43:27] :) [17:43:27] :) [17:43:34] anyway, I am heading back home [17:43:34] anyway, I am heading back home [17:43:46] thank you for all the tweaks and puppet dance! [17:43:46] thank you for all the tweaks and puppet dance! [17:43:48] i am applying the httpd config right now. done [17:43:48] i am applying the httpd config right now. done [17:43:56] you are welcome. ttyl then :) [17:43:56] you are welcome. ttyl then :) [17:44:03] ah [17:44:03] ah [17:45:10] mutante: might need apache to be restarted [17:45:10] mutante: might need apache to be restarted [17:45:49] done, but: [17:45:49] done, but: [17:45:49] [doc1001:~] $ sudo apache2ctl -S [17:45:49] [doc1001:~] $ sudo apache2ctl -S [17:45:50] [Wed Dec 19 17:45:34.425679 2018] [core:warn] [pid 28972:tid 139967184147648] AH00117: Ignoring deprecated use of DefaultType in line 57 of /etc/apache2/sites-enabled/50-doc-wikimedia-org.conf. [17:45:50] [Wed Dec 19 17:45:34.425679 2018] [core:warn] [pid 28972:tid 139967184147648] AH00117: Ignoring deprecated use of DefaultType in line 57 of /etc/apache2/sites-enabled/50-doc-wikimedia-org.conf. [17:46:04] *:80 doc.wikimedia.org (/etc/apache2/sites-enabled/50-doc-wikimedia-org.conf:8) [17:46:04] *:80 doc.wikimedia.org (/etc/apache2/sites-enabled/50-doc-wikimedia-org.conf:8) [17:46:11] there is the vhost [17:46:11] there is the vhost [17:46:42] i also just get the default though when sending host header with curl.. uhmm.. [17:46:42] i also just get the default though when sending host header with curl.. uhmm.. [17:47:24] I get the dummy page instead :( [17:47:24] I get the dummy page instead :( [17:47:52] ah, hashar, i know [17:47:52] ah, hashar, i know [17:47:55] i had the same issue on another host [17:47:55] i had the same issue on another host [17:48:00] we should not try to use localhost :) [17:48:00] we should not try to use localhost :) [17:48:05] use the actual IP [17:48:05] use the actual IP [17:48:12] curl -H "Host: doc.wikimedia.org" http://10.64.0.142 [17:48:12] curl -H "Host: doc.wikimedia.org" http://10.64.0.142 [17:48:12] require_once __DIR__ . '/../../../shared/DocPage.php'; [17:48:12] require_once __DIR__ . '/../../../shared/DocPage.php'; [17:48:33] ohhhh [17:48:33] ohhhh [17:48:35] needs PHP next :)2 [17:48:35] needs PHP next :)2 [17:49:13] that is stuff that we are in the process of changing (PHP config httpd php 7.2 php-fpm ...) [17:49:13] that is stuff that we are in the process of changing (PHP config httpd php 7.2 php-fpm ...) [17:49:14] and there is a bit more mess I need to sort out [17:49:14] and there is a bit more mess I need to sort out [17:49:23] i can work on that part in a bit [17:49:23] i can work on that part in a bit [17:49:27] namely that the docroot is in integration/docroot.git and deployed by a jenkins job [17:49:27] namely that the docroot is in integration/docroot.git and deployed by a jenkins job [17:49:31] but that should be ported to scap [17:49:31] but that should be ported to scap [17:49:33] bah [17:49:33] bah [17:49:44] so yeah more stuff I have to complete :) [17:49:44] so yeah more stuff I have to complete :) [17:49:46] ah, ok, so maybe we should just make a copy of the template right now anyways [17:49:46] ah, ok, so maybe we should just make a copy of the template right now anyways [17:49:52] and then change it in new place [17:49:52] and then change it in new place [17:50:04] yeah if you want [17:50:04] yeah if you want [17:50:14] I migth also change paths a bit [17:50:14] I migth also change paths a bit [17:50:22] ok [17:50:22] ok [17:50:25] to use /srv/docroot as abase path instead of just /srv [17:50:25] to use /srv/docroot as abase path instead of just /srv [17:50:30] but that will be for tomorrow [17:50:30] but that will be for tomorrow [17:50:34] :) [17:50:34] :) [17:50:35] i will do that and make PHP work after coffee [17:50:35] i will do that and make PHP work after coffee [17:50:46] awesome thanks [17:50:46] awesome thanks [17:50:56] going to have dinner now [17:50:56] going to have dinner now [17:53:09] 10Continuous-Integration-Config, 10ExternalGuidance, 10User-MarcoAurelio: Add CI support to ExternalGuidance - https://phabricator.wikimedia.org/T212286 (10MarcoAurelio) 05Open→03Resolved CI set and jobs pass :) [17:53:09] 10Continuous-Integration-Config, 10ExternalGuidance, 10User-MarcoAurelio: Add CI support to ExternalGuidance - https://phabricator.wikimedia.org/T212286 (10MarcoAurelio) 05Open→03Resolved CI set and jobs pass :) [18:05:11] !log Created mediawiki/extensions/ExternalGuidance.git repos on Gerrit, set-up mirrors on Diffusion and GitHub; per mediawiki.org request. [18:05:11] !log Created mediawiki/extensions/ExternalGuidance.git repos on Gerrit, set-up mirrors on Diffusion and GitHub; per mediawiki.org request. [18:05:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:05:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:18:16] mutante: ping [18:18:16] mutante: ping [18:21:26] Hauskatze: pong (https://blogs.gnome.org/markmc/2014/02/20/naked-pings/) [18:21:26] Hauskatze: pong (https://blogs.gnome.org/markmc/2014/02/20/naked-pings/) [18:22:17] mutante: I'm having troubles with the mediawiki mailer [18:22:17] mutante: I'm having troubles with the mediawiki mailer [18:22:29] it looks it is not working for beta? [18:22:29] it looks it is not working for beta? [18:22:44] perhaps when we switched servers we forgot to reconfigure it? [18:22:44] perhaps when we switched servers we forgot to reconfigure it? [18:23:27] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10MarcoAurelio) [18:23:27] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10MarcoAurelio) [18:23:29] Hauskatze: sorry, i don't know about this. switched servers? [18:23:29] Hauskatze: sorry, i don't know about this. switched servers? [18:23:46] from deployment-tin to deployment-deploy01, etc. [18:23:46] from deployment-tin to deployment-deploy01, etc. [18:24:03] oh, are you sure it worked before? [18:24:03] oh, are you sure it worked before? [18:24:25] let me try find a ticket.. hmm [18:24:25] let me try find a ticket.. hmm [18:24:59] Hauskatze: i know that this happened https://phabricator.wikimedia.org/T41785 [18:24:59] Hauskatze: i know that this happened https://phabricator.wikimedia.org/T41785 [18:25:07] mutante: yes I'm sure it worked before [18:25:07] mutante: yes I'm sure it worked before [18:25:43] Hauskatze: ok, so the new SMTP smarthost for cloud VPS.. it seems it was switched to that on Nov 13 [18:25:43] Hauskatze: ok, so the new SMTP smarthost for cloud VPS.. it seems it was switched to that on Nov 13 [18:25:48] so maybe related to that [18:25:48] so maybe related to that [18:26:08] mutante: so apparently they ain't working [18:26:08] mutante: so apparently they ain't working [18:26:14] per https://phabricator.wikimedia.org/T41785#4736149 [18:26:14] per https://phabricator.wikimedia.org/T41785#4736149 [18:26:16] I'm trying to dig in the var/logs [18:26:16] I'm trying to dig in the var/logs [18:27:01] I guess I'll have to reset the password by hand :| [18:27:01] I guess I'll have to reset the password by hand :| [18:27:15] Hauskatze: try to find out which mail servers it's trying to use. these should be the new ones: mx-out0[12].wmflabs.org [18:27:15] Hauskatze: try to find out which mail servers it's trying to use. these should be the new ones: mx-out0[12].wmflabs.org [18:27:51] I'll check logstash [18:27:51] I'll check logstash [18:30:15] nothing there [18:30:15] nothing there [18:36:51] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10dduvall) >>! In T212251#4834349, @akosiaris wrote: >>>! In T212251#4834325, @hashar wrote: >>>>! In T212251#4833906, @akosiaris wrote: >>> As S... [18:36:51] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10serviceops: Allow access to blubberoid.discovery.wmnet:8748 - https://phabricator.wikimedia.org/T212251 (10dduvall) >>! In T212251#4834349, @akosiaris wrote: >>>! In T212251#4834325, @hashar wrote: >>>>! In T212251#4833906, @akosiaris wrote: >>> As S... [18:38:12] Hauskatze: /var/log/mail.log on the instance? [18:38:12] Hauskatze: /var/log/mail.log on the instance? [18:38:29] mutante: on which server? [18:38:29] mutante: on which server? [18:38:39] fluorine or deploy01? [18:38:39] fluorine or deploy01? [18:38:41] Hauskatze: the one that you want to send out mail [18:38:41] Hauskatze: the one that you want to send out mail [18:38:55] good question, cause I don't know [18:38:55] good question, cause I don't know [18:39:08] fluorine is something that once existed in prod but not anymore [18:39:08] fluorine is something that once existed in prod but not anymore [18:39:24] I think its deployment-mx02 [18:39:24] I think its deployment-mx02 [18:39:31] I see some logstash errors [18:39:31] I see some logstash errors [18:39:35] that sounds like it's the mail server [18:39:35] that sounds like it's the mail server [18:39:36] some certcentral stuff [18:39:36] some certcentral stuff [18:39:41] i meant the one that is sending TO the mail server [18:39:41] i meant the one that is sending TO the mail server [18:39:58] Could not retrieve file metadata for puppet://deployment-certcentral03.deployment-prep.eqiad.wmflabs/acmedata/mx/ec-prime256v1.key: Failed to open TCP connection to deployment-certcentral03.deployment-prep.eqiad.wmflabs:8140 (getaddrinfo: Name or service not known) [18:39:58] Could not retrieve file metadata for puppet://deployment-certcentral03.deployment-prep.eqiad.wmflabs/acmedata/mx/ec-prime256v1.key: Failed to open TCP connection to deployment-certcentral03.deployment-prep.eqiad.wmflabs:8140 (getaddrinfo: Name or service not known) [18:40:42] Hauskatze: that's about https://phabricator.wikimedia.org/tag/certcentral/ and should be unrelated to email [18:40:42] Hauskatze: that's about https://phabricator.wikimedia.org/tag/certcentral/ and should be unrelated to email [18:41:01] zeljkof: around? [18:41:01] zeljkof: around? [18:41:02] you might want to ping the 2 watcher son that [18:41:02] you might want to ping the 2 watcher son that [18:44:08] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) shell access for existing groups contint-admins and contint-users has been granted (same access people had be... [18:44:08] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) shell access for existing groups contint-admins and contint-users has been granted (same access people had be... [18:45:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10serviceops, 10Developer-Wishlist (2017), and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Dzahn) [18:45:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10serviceops, 10Developer-Wishlist (2017), and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Dzahn) [18:46:34] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10MarcoAurelio) @herron - not sure if related to T41785; also @aborrero due to being in wmflabs realm. [18:46:34] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10MarcoAurelio) @herron - not sure if related to T41785; also @aborrero due to being in wmflabs realm. [19:38:12] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10herron) What outbound smtp server is currently being used by the beta cluster? [19:38:12] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10herron) What outbound smtp server is currently being used by the beta cluster? [19:40:33] hexmode: sorry just saw your ping, on the phone [19:40:33] hexmode: sorry just saw your ping, on the phone [19:44:24] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10greg) [19:44:24] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10greg) [19:44:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Mail, 10MediaWiki-Email, 10Regression: betalabs email is not working - https://phabricator.wikimedia.org/T212249 (10greg) [19:44:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Mail, 10MediaWiki-Email, 10Regression: betalabs email is not working - https://phabricator.wikimedia.org/T212249 (10greg) [19:44:45] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) I think I set beta's wikimail stuff to go via deployment-mx02, will look into it [19:44:45] 10Beta-Cluster-Infrastructure: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) I think I set beta's wikimail stuff to go via deployment-mx02, will look into it [19:44:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Discovery-Search, 10Elasticsearch: Set up data storage to collect loosely structured data from CI - https://phabricator.wikimedia.org/T211904 (10mmodell) @ebjune: Thanks for the heads-up. I definitely want EBernhardson to weigh... [19:44:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Discovery-Search, 10Elasticsearch: Set up data storage to collect loosely structured data from CI - https://phabricator.wikimedia.org/T211904 (10mmodell) @ebjune: Thanks for the heads-up. I definitely want EBernhardson to weigh... [19:44:55] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10greg) [19:44:55] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10greg) [19:45:32] 10Continuous-Integration-Infrastructure, 10Technical-Debt: Clear /srv/.git on contint1001 - https://phabricator.wikimedia.org/T149924 (10hashar) [19:45:32] 10Continuous-Integration-Infrastructure, 10Technical-Debt: Clear /srv/.git on contint1001 - https://phabricator.wikimedia.org/T149924 (10hashar) [19:46:32] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) 05Open→03Resolved Yes, the VM has been created, basic role has been created, users added, httpd installed... [19:46:32] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests, 10Patch-For-Review: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) 05Open→03Resolved Yes, the VM has been created, basic role has been created, users added, httpd installed... [19:46:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10serviceops, 10Developer-Wishlist (2017), and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Dzahn) [19:46:35] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10serviceops, 10Developer-Wishlist (2017), and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Dzahn) [19:47:05] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) [19:47:05] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10Dzahn) [19:48:54] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) Thank you for the quick spinning of the instance as well as all the preliminary puppet work. Much appreciated :) [19:48:54] 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, 10vm-requests: eqiad: 1 VM request for doc.wikimedia.org - https://phabricator.wikimedia.org/T211974 (10hashar) Thank you for the quick spinning of the instance as well as all the preliminary puppet work. Much appreciated :) [19:49:16] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) From `deployment-mx02:/var/log/exim4/mainlog`: `2018-12-19 19:45:50 H=deployment-mediawiki-07.deployment-prep.eqiad.wmflabs [172.16.4.119]:45122... [19:49:16] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) From `deployment-mx02:/var/log/exim4/mainlog`: `2018-12-19 19:45:50 H=deployment-mediawiki-07.deployment-prep.eqiad.wmflabs [172.16.4.119]:45122... [20:02:39] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10MarcoAurelio) [20:02:39] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10MarcoAurelio) [20:03:10] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) Well that's weird, dunno why puppet's autoupdater was stuck while rebasing seemed to work fine: ` root@deployment-puppetmaster03:/var/lib/git/ope... [20:03:10] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) Well that's weird, dunno why puppet's autoupdater was stuck while rebasing seemed to work fine: ` root@deployment-puppetmaster03:/var/lib/git/ope... [20:24:10] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) (None of that seemed to have any effect on -mx02.) [20:24:10] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) (None of that seemed to have any effect on -mx02.) [20:31:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) [20:31:09] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) [20:46:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) Deployed and it looks fine so far. [20:46:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T206663 (10hashar) Deployed and it looks fine so far. [20:49:28] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) So I think this broke during the eqiad1-r migration. /etc/exim4/exim4.conf: `hostlist wikimedia_nets = <; 91.198.174.0/24 ; 208.80.152.0/22 ; 262... [20:49:28] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) So I think this broke during the eqiad1-r migration. /etc/exim4/exim4.conf: `hostlist wikimedia_nets = <; 91.198.174.0/24 ; 208.80.152.0/22 ; 262... [21:02:07] (03PS1) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:02:07] (03PS1) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:08:57] (03CR) 10Thcipriani: "feedback inline" (033 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:08:57] (03CR) 10Thcipriani: "feedback inline" (033 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:12:00] (03PS2) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:12:00] (03PS2) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:13:09] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) `hostlist wikimedia_nets = <; <%= scope.lookupvar('network::constants::all_networks').join(" ; ") %>` `hostlist relay_from_hosts = <; @[] ; 127.0... [21:13:09] 10Beta-Cluster-Infrastructure, 10Mail, 10MediaWiki-Email: Beta Cluster mailer not sending emails apparently - https://phabricator.wikimedia.org/T212327 (10Krenair) `hostlist wikimedia_nets = <; <%= scope.lookupvar('network::constants::all_networks').join(" ; ") %>` `hostlist relay_from_hosts = <; @[] ; 127.0... [21:19:40] (03CR) 10Hashar: scap configuration for deployment (033 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:19:40] (03CR) 10Hashar: scap configuration for deployment (033 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:19:54] (03PS3) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:19:54] (03PS3) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:24:37] (03CR) 10Thcipriani: scap configuration for deployment (032 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:24:37] (03CR) 10Thcipriani: scap configuration for deployment (032 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:29:36] (03PS4) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:29:36] (03PS4) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) [21:29:49] mobileapps-deploy-npm-node-6-docker is failing because of what seems like a problem with the CI setup [21:29:49] mobileapps-deploy-npm-node-6-docker is failing because of what seems like a problem with the CI setup [21:30:09] not sure exactly what yet; spelunking in the console logs [21:30:09] not sure exactly what yet; spelunking in the console logs [21:30:45] https://integration.wikimedia.org/ci/job/mobileapps-deploy-npm-node-6-docker/ [21:30:45] https://integration.wikimedia.org/ci/job/mobileapps-deploy-npm-node-6-docker/ [21:31:06] (03CR) 10Hashar: scap configuration for deployment (032 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:31:06] (03CR) 10Hashar: scap configuration for deployment (032 comments) [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [21:35:16] maintenance-disconnect-full-disks build 30345 integration-slave-jessie-1003 (/srv: 95%): OFFLINE due to disk space [21:35:16] maintenance-disconnect-full-disks build 30345 integration-slave-jessie-1003 (/srv: 95%): OFFLINE due to disk space [21:36:05] hashar: where is "shared/DocPage.php" located ? [21:36:05] hashar: where is "shared/DocPage.php" located ? [21:36:23] mutante: in integration/docroot.git which is deployed by Jenkins [21:36:23] mutante: in integration/docroot.git which is deployed by Jenkins [21:36:33] I am crafting a patch to have it deployed via scap instead [21:36:33] I am crafting a patch to have it deployed via scap instead [21:36:33] ) [21:36:33] ) [21:37:01] hashar: aha! current status is you dont get something from / URL because PHP Fatal error: require_once( [21:37:01] hashar: aha! current status is you dont get something from / URL because PHP Fatal error: require_once( [21:37:08] that explains and i will stop wondering [21:37:08] that explains and i will stop wondering [21:37:14] i claim PHP works :) [21:37:14] i claim PHP works :) [21:37:42] well that is good [21:37:42] well that is good [21:37:44] PHP works! :) [21:37:44] PHP works! :) [21:37:54] yea, as the Fatal error shows :) [21:37:54] yea, as the Fatal error shows :) [21:38:32] now that there is the file handler to tell it to use php-fpm for .php files.. ack [21:38:32] now that there is the file handler to tell it to use php-fpm for .php files.. ack [21:38:43] be back in a bit [21:38:43] be back in a bit [21:40:15] hashar: scap might be overkill for a few php files? we can just git clone from puppet too [21:40:15] hashar: scap might be overkill for a few php files? we can just git clone from puppet too [21:40:28] be back after lunch [21:40:28] be back after lunch [21:55:29] mutante: with git::clone we cant really control the deployment :) [21:55:29] mutante: with git::clone we cant really control the deployment :) [21:55:55] I also want to be able to rollback easily without having to wait for puppet to kick in [21:55:55] I also want to be able to rollback easily without having to wait for puppet to kick in [21:58:27] thcipriani: and eventually I have added scap::target to my profile https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480832/ ! :) [21:58:27] thcipriani: and eventually I have added scap::target to my profile https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/480832/ ! :) [22:00:14] maintenance-disconnect-full-disks build 30350 integration-slave-jessie-1003: OFFLINE due to disk space [22:00:14] maintenance-disconnect-full-disks build 30350 integration-slave-jessie-1003: OFFLINE due to disk space [22:03:01] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Upgrade to Gerrit 2.16.2 - https://phabricator.wikimedia.org/T200739 (10Paladox) [22:03:01] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Upgrade to Gerrit 2.16.2 - https://phabricator.wikimedia.org/T200739 (10Paladox) [22:03:38] (03PS1) 10MarcoAurelio: WIP: Add sec-check for ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480877 [22:03:38] (03PS1) 10MarcoAurelio: WIP: Add sec-check for ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480877 [22:05:02] (03PS2) 10MarcoAurelio: Add extension-seccheck for ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480877 [22:05:02] (03PS2) 10MarcoAurelio: Add extension-seccheck for ExternalGuidance [integration/config] - 10https://gerrit.wikimedia.org/r/480877 [22:05:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10serviceops, 10Developer-Wishlist (2017), and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10hashar) The scap config for integration/docroot is: * https://gerrit.wikimedi... [22:05:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10serviceops, 10Developer-Wishlist (2017), and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10hashar) The scap config for integration/docroot is: * https://gerrit.wikimedi... [22:12:26] hashar: using git::clone does not mean we have to also ensure "latest". ensure "present" means you still control the deployment [22:12:26] hashar: using git::clone does not mean we have to also ensure "latest". ensure "present" means you still control the deployment [22:12:35] you would git pull when you decide [22:12:35] you would git pull when you decide [22:13:40] thcipriani: I thought about depliyng integration/docroot with scap that is Apache DocumentRoot [22:13:40] thcipriani: I thought about depliyng integration/docroot with scap that is Apache DocumentRoot [22:13:59] but then I get the CI job to rsync material under the subdir org/wikimedia/doc which then is used for doc.wikimedia.ogr [22:13:59] but then I get the CI job to rsync material under the subdir org/wikimedia/doc which then is used for doc.wikimedia.ogr [22:14:40] hrm [22:14:40] hrm [22:14:43] and seems like scap creates a fresh brand new checkout on the target [22:14:43] and seems like scap creates a fresh brand new checkout on the target [22:14:49] thus the sub directory will be gone [22:14:49] thus the sub directory will be gone [22:14:53] indeed [22:14:53] indeed [22:15:15] unless you could keep that directory in a different place and symlink it back in at the end of deployment [22:15:15] unless you could keep that directory in a different place and symlink it back in at the end of deployment [22:15:46] not really :/ [22:15:46] not really :/ [22:16:05] integration/docroot.git has bunch of boilerplate into org/wikimedia/doc/ [22:16:05] integration/docroot.git has bunch of boilerplate into org/wikimedia/doc/ [22:16:18] and the published docs are directly into it [22:16:18] and the published docs are directly into it [22:16:33] so maybe it is easier to just git::clone ensure => latest as suggested by mutante [22:16:33] so maybe it is easier to just git::clone ensure => latest as suggested by mutante [22:16:43] with some rule to let us manually pull if need be [22:16:43] with some rule to let us manually pull if need be [22:16:54] for one server it might be easier anyway [22:16:54] for one server it might be easier anyway [22:17:20] I felt it was a lowhanging / easy way to get more familiar with scap : [22:17:20] I felt it was a lowhanging / easy way to get more familiar with scap : [22:17:20] ) [22:17:20] ) [22:17:29] but docroot is probably a bad example hehe [22:17:29] but docroot is probably a bad example hehe [22:25:13] maintenance-disconnect-full-disks build 30355 integration-slave-jessie-1003: OFFLINE due to disk space [22:25:13] maintenance-disconnect-full-disks build 30355 integration-slave-jessie-1003: OFFLINE due to disk space [22:26:23] (03Abandoned) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [22:26:23] (03Abandoned) 10Hashar: scap configuration for deployment [integration/docroot] - 10https://gerrit.wikimedia.org/r/480828 (https://phabricator.wikimedia.org/T137890) (owner: 10Hashar) [22:50:13] maintenance-disconnect-full-disks build 30360 integration-slave-jessie-1003: OFFLINE due to disk space [22:50:13] maintenance-disconnect-full-disks build 30360 integration-slave-jessie-1003: OFFLINE due to disk space [23:01:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Technical-Debt: Clear /srv/.git on contint1001 - https://phabricator.wikimedia.org/T149924 (10hashar) a:03hashar Will be addressed as we migrate the documentation hosting to a new host. I am intending to... [23:01:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Technical-Debt: Clear /srv/.git on contint1001 - https://phabricator.wikimedia.org/T149924 (10hashar) a:03hashar Will be addressed as we migrate the documentation hosting to a new host. I am intending to... [23:15:13] maintenance-disconnect-full-disks build 30365 integration-slave-jessie-1003: OFFLINE due to disk space [23:15:13] maintenance-disconnect-full-disks build 30365 integration-slave-jessie-1003: OFFLINE due to disk space [23:27:44] !log integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online [23:27:44] !log integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online [23:27:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:27:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:40:31] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 4 others: Timeout exceeded for WikibaseReferenceOnProtectedPage Selenium test - https://phabricator.wikimedia.org/T210929 (10Krinkle) Has this flaky test been fixed/disabled, or is there some re... [23:40:31] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 4 others: Timeout exceeded for WikibaseReferenceOnProtectedPage Selenium test - https://phabricator.wikimedia.org/T210929 (10Krinkle) Has this flaky test been fixed/disabled, or is there some re... [23:41:04] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 2 others: Timeout exceeded for WikibaseReferenceOnProtectedPage Selenium test - https://phabricator.wikimedia.org/T210929 (10Krinkle) [23:41:04] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Campsite, and 2 others: Timeout exceeded for WikibaseReferenceOnProtectedPage Selenium test - https://phabricator.wikimedia.org/T210929 (10Krinkle)