[00:09:42] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10mmodell) 05Openβ†’03Resolved [00:09:58] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10mmodell) @RhinosF1 it's been deployed now. [00:13:52] (03CR) 10Thcipriani: [C: 03+2] sonar-scanner: Use relative paths and mount to /workspace/src [integration/config] - 10https://gerrit.wikimedia.org/r/508929 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [00:15:45] (03Merged) 10jenkins-bot: sonar-scanner: Use relative paths and mount to /workspace/src [integration/config] - 10https://gerrit.wikimedia.org/r/508929 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [00:19:08] !log clean docker images on contint1001 [00:19:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:19:41] !log updating docker images on contint1001 for https://gerrit.wikimedia.org/r/508929 [00:19:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:22:15] (03CR) 10Thcipriani: [C: 03+2] Clear MessageBlobStore after syncing i18n data [tools/scap] - 10https://gerrit.wikimedia.org/r/508488 (https://phabricator.wikimedia.org/T222539) (owner: 10Catrope) [00:25:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.039 second response time [00:25:29] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/508488 (https://phabricator.wikimedia.org/T222539) (owner: 10Catrope) [00:25:31] (03Merged) 10jenkins-bot: Clear MessageBlobStore after syncing i18n data [tools/scap] - 10https://gerrit.wikimedia.org/r/508488 (https://phabricator.wikimedia.org/T222539) (owner: 10Catrope) [00:26:07] (03CR) 10jenkins-bot: Clear MessageBlobStore after syncing i18n data [tools/scap] - 10https://gerrit.wikimedia.org/r/508488 (https://phabricator.wikimedia.org/T222539) (owner: 10Catrope) [00:27:06] 10Release-Engineering-Team, 10Scap, 10MediaWiki-ResourceLoader, 10Patch-For-Review, and 2 others: Scap deployments are not purging MessageBlobStore (was: Stale localized messages) - https://phabricator.wikimedia.org/T222539 (10thcipriani) >>! In T222539#5169146, @gerritbot wrote: > Change 508488 **merged**... [00:36:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [01:56:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.023 second response time [02:02:03] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [02:25:06] (03PS10) 10Kosta Harlan: Generate junit.xml for sonar-scanner's usage [integration/config] - 10https://gerrit.wikimedia.org/r/508019 (https://phabricator.wikimedia.org/T218598) [02:25:27] (03PS11) 10Kosta Harlan: Generate junit.xml for sonar-scanner's usage [integration/config] - 10https://gerrit.wikimedia.org/r/508019 (https://phabricator.wikimedia.org/T208522) [02:30:32] (03PS28) 10Kosta Harlan: Establish codehealth pipeline, enable for three extensions [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) [02:31:25] (03CR) 10Kosta Harlan: Establish codehealth pipeline, enable for three extensions (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/502606 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan) [02:42:04] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.023 second response time [02:53:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [03:18:03] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.025 second response time [04:59:42] PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [05:05:13] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:30:27] Yippee, build fixed! [05:30:28] Project mediawiki-core-code-coverage-docker build #4238: 09FIXED in 2 hr 30 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-docker/4238/ [05:30:47] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10RhinosF1) Thanks @mmodell [06:25:56] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out) [08:16:42] 10Release-Engineering-Team, 10Scap, 10MediaWiki-ResourceLoader, 10Patch-For-Review, and 2 others: Scap deployments are not purging MessageBlobStore (was: Stale localized messages) - https://phabricator.wikimedia.org/T222539 (10jeblad) Thanks to everyone! :D [08:17:04] !log remove mediawiki memcached nutcracker config from deployment-prep (should be unused) - T214275 [08:17:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:17:07] T214275: Consider removing the last traces of nutcracker in Mediawiki configs - https://phabricator.wikimedia.org/T214275 [08:17:17] please let me know if this causes any issue --^ [09:05:05] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.035 second response time [09:11:04] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [09:40:06] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Joe) There is a simple solution to run services that are now on k8s on deployment-prep: - Creat... [09:41:41] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Joe) Please also note you can run multiple services on the same VM if you really want to, it's e... [10:34:56] 10Phabricator: Specific existing project tag not found by Phabricator's project search - https://phabricator.wikimedia.org/T222870 (10Aklapper) [11:02:50] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Will the service run into any differences in its environment due to being run with role... [11:09:26] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, and 2 others: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10WMDE-leszek) [11:09:40] 10Gerrit, 10Shape Expressions, 10Wikidata, 10Patch-For-Review, and 2 others: Replace WikibaseSchema repository content with message pointing to EntitySchema - https://phabricator.wikimedia.org/T222192 (10WMDE-leszek) 05Openβ†’03Resolved [11:49:30] 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Joe) I don't really know why stretch won't work, are we sure that's the case? [11:53:01] 10Gerrit, 10Repository-Admins, 10Shape Expressions, 10Wikidata, and 2 others: rename repository for WikibaseSchema - https://phabricator.wikimedia.org/T221946 (10WMDE-leszek) [13:05:56] We have a postmerge job in zuul for 15 hours and 8 minutes: https://integration.wikimedia.org/zuul/ [13:06:55] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10Aklapper) For the records, I sent a heads-up announcement to `wikitech-l@` at https://lists.wikimedia.org/pipermail/wikitech-l/2019-May/092063.html and also pos... [13:07:09] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10Aklapper) [13:34:33] (03CR) 10Zfilipin: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:35:22] (03CR) 10jerkins-bot: [V: 04-1] Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:43:21] (03PS7) 10Zfilipin: Create selenium-daily-beta-ORES Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) [13:44:22] (03CR) 10Zfilipin: "PS7 fixes the error introduced in PS6 πŸ€¦β€β™‚οΈ" [integration/config] - 10https://gerrit.wikimedia.org/r/460517 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:44:38] (03PS6) 10Zfilipin: Create selenium-daily-beta-TwoColConflict Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) [13:44:47] (03PS7) 10Zfilipin: Create selenium-daily-beta-TwoColConflict Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460560 (https://phabricator.wikimedia.org/T188742) [13:45:27] (03PS6) 10Zfilipin: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) [13:50:12] (03PS7) 10Zfilipin: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) [13:51:08] (03CR) 10Zfilipin: "PS5 is a rebase that introduced a problem πŸ€¦β€β™‚οΈ" [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:55:30] (03CR) 10Zfilipin: "PS6 is a rebase, the problem is fixed in parent commit." [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:56:21] (03CR) 10Zfilipin: "PS7 removes the commit from previous relation chain and makes it a stand-alone commit." [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [13:59:24] Project beta-code-update-eqiad build #245896: 04FAILURE in 1 min 32 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245896/ [14:03:01] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Ottomata) I believe the VM has to be Jessie atm, unfortuntely. Can't remember exactly why. Sin... [14:03:56] Project beta-code-update-eqiad build #245897: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245897/ [14:05:18] 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Ottomata) I can't remember exactly, but it didn't when I tried months ago. We should try again. [14:11:57] (03CR) 10Zfilipin: Create selenium-daily-beta-WikibaseLexeme Jenkins job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [14:13:56] Project beta-code-update-eqiad build #245898: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245898/ [14:14:43] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10akosiaris) > > Since the docker container will be the same as the one running in production, I... [14:16:06] RECOVERY - Mathoid on deployment-mathoid is OK: HTTP OK: HTTP/1.1 200 OK - 925 bytes in 0.511 second response time [14:19:35] 10Gerrit: Unable to login to gerrit with my credential due to duplication entry - https://phabricator.wikimedia.org/T222715 (10Aklapper) Tentatively CC'ing @thcipriani here [14:21:13] (03CR) 10Zfilipin: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [14:22:05] PROBLEM - Mathoid on deployment-mathoid is CRITICAL: connect to address 172.16.5.73 and port 10042: Connection refused [14:22:36] (03CR) 10Zfilipin: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [14:23:10] (03CR) 10Zfilipin: [C: 03+2] Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [14:23:56] Project beta-code-update-eqiad build #245899: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245899/ [14:25:53] (03Merged) 10jenkins-bot: Create selenium-daily-beta-WikibaseLexeme Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/460527 (https://phabricator.wikimedia.org/T188742) (owner: 10Zfilipin) [14:28:22] 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10WMDE-leszek) Hey @akosiaris and @mobrovac we've been wondering if you had a chance to look into our service again. As reported... [14:29:17] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:33:56] Project beta-code-update-eqiad build #245900: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245900/ [14:34:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:36:16] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:36:54] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:40:24] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:41:45] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:42:32] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:43:07] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10Patch-For-Review, 10User-zeljkofilipin: Run tests daily targeting beta cluster for all repositories with Selenium tests - https://phabricator.wikimedia.org/T188742 (10zeljkofilipin) [14:43:56] Project beta-code-update-eqiad build #245901: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245901/ [14:45:22] 15:43:43 INFO:mwextupdate:running: git submodule update --init --recursive [14:45:22] 15:43:53 error: no such remote ref e404a34d7727ad1839db63f9de9a9e65e7062a79 [14:45:22] 15:43:53 Fetched in submodule path 'EntitySchema', but it did not contain e404a34d7727ad1839db63f9de9a9e65e7062a79. Direct fetching of that commit failed. [14:45:25] I see wikidata has broken beta [14:47:12] https://github.com/wikimedia/mediawiki-extensions-EntitySchema/commit/e404a34d7727ad1839db63f9de9a9e65e7062a79 [14:47:17] wat [14:53:56] Project beta-code-update-eqiad build #245902: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245902/ [15:03:55] Project beta-code-update-eqiad build #245903: 04STILL FAILING in 54 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245903/ [15:13:59] Project beta-code-update-eqiad build #245904: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245904/ [15:16:11] well that's confusing. [15:17:44] looks like that commit doesn't exist in actual entityschema logs: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikibaseSchema/+log/HEAD vs https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/EntitySchema/+log/HEAD -- github redirects one to the other [15:18:13] the parent of that commit exists in EntitySchema, which makes sense if you intend to continue using that repo [15:21:26] 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Joe) Ok so I can now confirm: this all works on stretch. I've successfully installed `deployment-docker-mathoid01` with stretch and managed to make mathoid run there... [15:21:56] 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Joe) The only caveat is that apparently you need to rrun puppet once, run apt-get update, run puppet again to make it work. [15:22:08] sorry :/ [15:22:30] thcipriani: I will fix it [15:22:43] Amir1: thanks! [15:23:57] Project beta-code-update-eqiad build #245905: 04STILL FAILING in 55 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245905/ [15:24:58] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Joe) >>! In T220235#5169739, @Krenair wrote: > Will the service run into any differences in its... [15:26:12] 10Beta-Cluster-Infrastructure, 10serviceops: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Joe) The way to go for such things is to use `role::beta::docker_services` on a fresh VM. I've already created deployment-docker-mathoid01 that should replace the old mathoid ser... [15:30:47] thcipriani: I pushed the patch, I hope it works [15:32:23] Amir1: I think that should do it, we'll see if it makes beta-code-update happy again [15:33:07] I'm here to distribute kudos. After two terrible hours of trying to install and run mw-phan-seccheck locally, in the end I was able use the docker image to reproduce the errors reported by WMF CI, and debug my code. [15:33:58] awight: nice :) [15:34:30] Yippee, build fixed! [15:34:30] Project beta-code-update-eqiad build #245906: 09FIXED in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/245906/ [15:34:44] ^ Amir1 thank you for the patch! [15:35:34] no, sorry for breaking it [15:47:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): Upgrade python-pbr on contint1001 / contint2001 and restart Zuul process - https://phabricator.wikimedia.org/T222659 (10greg) p:05Triageβ†’03Normal [15:49:33] 10Release-Engineering-Team (Kanban), 10User-MModell: Talk with Timo and Fillipo about grafana and sentury for LM ("logging, monitoring, metrics") - https://phabricator.wikimedia.org/T222638 (10greg) p:05Triageβ†’03Normal [15:51:06] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10dev-images, 10local-charts: Move dev-images PHP + Apache image from mod_php to php-fpm - https://phabricator.wikimedia.org/T222494 (10greg) p:05Triageβ†’03Normal [15:51:11] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10Documentation: Improve documentation on Docker-based development environments for new developers - https://phabricator.wikimedia.org/T217614 (10greg) p:05Triageβ†’03Normal [15:51:13] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts: Add tests to local-charts / configure local-charts for CI - https://phabricator.wikimedia.org/T217868 (10greg) p:05Triageβ†’03Normal [16:03:24] 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10Tgr) Phabricator comes with a wide array of tools, trying to cover everything that you would expect from a forge / communication hub / project management softwa... [16:12:08] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [16:13:26] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10MW-1.34-notes (1.34.0-wmf.5; 2019-05-14): Stop using jsonlint (as it's abandonware) and instead use eslint-plugin-json for the linting - https://phabricator.wikimedia.org/T220036 (10Jdforrester-WMF) [16:42:09] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.026 second response time [16:48:08] PROBLEM - Citoid on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 1970: Connection refused [16:58:48] PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Discovery-Search: quibble-vendor-mysql-hhvm-docker for WikibaseCirrusSearch takes over 40 minutes - https://phabricator.wikimedia.org/T222757 (10debt) Moving this to our #watching column for now :) [17:08:36] RECOVERY - Content Translation Server on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.026 second response time [17:13:08] RECOVERY - Citoid on deployment-sca02 is OK: HTTP OK: HTTP/1.1 200 OK - 921 bytes in 0.024 second response time [17:23:19] (03PS1) 10Umherirrender: [TranslationNotifications] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/509121 [17:27:15] (03CR) 10Zfilipin: [C: 03+2] Send e-mail notification to Ephemeralwaves if a job fails [integration/config] - 10https://gerrit.wikimedia.org/r/508350 (https://phabricator.wikimedia.org/T217051) (owner: 10Zfilipin) [17:31:12] (03Merged) 10jenkins-bot: Send e-mail notification to Ephemeralwaves if a job fails [integration/config] - 10https://gerrit.wikimedia.org/r/508350 (https://phabricator.wikimedia.org/T217051) (owner: 10Zfilipin) [17:40:32] (03PS2) 10Umherirrender: [EventLogging] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/507707 [17:41:23] (03CR) 10Umherirrender: [C: 04-1] "Changed from dependency change to add phan job" [integration/config] - 10https://gerrit.wikimedia.org/r/507707 (owner: 10Umherirrender) [17:58:31] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Who is able to make a call on this? Can I just merge it when nobody objected for such a long time?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/488561 (https://phabricator.wikimedia.org/T213861) (owner: 10Umherirrender) [18:12:14] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Testing, 10MW-1.34-notes (1.34.0-wmf.5; 2019-05-14): Stop using jsonlint (as it's abandonware) and instead use eslint-plugin-json for the linting - https://phabricator.wikimedia.org/T220036 (10Jdforrester-WMF) [18:14:43] RECOVERY - Puppet staleness on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [3600.0] [18:15:57] 10Release-Engineering-Team (Kanban), 10User-greg: Improve the effectiveness of #releng related workboards/process - https://phabricator.wikimedia.org/T222496 (10greg) tl;dr: I'm basically stealing some from the CPT boards/processes. :) [18:17:17] (03CR) 10Jforrester: [C: 03+1] "Let's land this?" [integration/config] - 10https://gerrit.wikimedia.org/r/502994 (owner: 10Hashar) [18:18:59] (03PS6) 10Thcipriani: pipeline: Directed graph execution model [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [18:19:05] (03CR) 10Thcipriani: [C: 03+2] "Looks great!" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [18:20:36] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [18:21:04] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [18:21:06] (03Merged) 10jenkins-bot: pipeline: Directed graph execution model [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [18:21:46] (03CR) 10jenkins-bot: pipeline: Directed graph execution model [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502917 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [18:25:47] (03CR) 10Umherirrender: "It is possible that now the dev dependency are part of the vendor folder?" [integration/config] - 10https://gerrit.wikimedia.org/r/508320 (https://phabricator.wikimedia.org/T189567) (owner: 10Hashar) [19:57:43] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10Jdforrester-WMF) [20:13:21] 10Release-Engineering-Team, 10Developer Productivity, 10Epic: FY201819 TEC12 Program – Developer Productivity - https://phabricator.wikimedia.org/T212449 (10brennen) [20:13:25] 10Release-Engineering-Team (Kanban), 10Developer Productivity, 10local-charts, 10Patch-For-Review: Script SSHFS setup in local-charts - https://phabricator.wikimedia.org/T218364 (10brennen) 05Openβ†’03Resolved We'll iterate on this, but it's working, at least. [20:22:22] 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10Catrope) Sounds sane to me; except that I think the scope might be extend... [20:26:12] 10Gerrit, 10Operations, 10cloud-services-team, 10serviceops: Change /r/p/ to /r/ on all hosts (where https://gerrit.wikimedia.org/r/p/ exists) - https://phabricator.wikimedia.org/T222093 (10Paladox) [20:33:54] 10Gerrit, 10Operations, 10cloud-services-team, 10serviceops: Change /r/p/ to /r/ on all hosts (where https://gerrit.wikimedia.org/r/p/ exists) - https://phabricator.wikimedia.org/T222093 (10Paladox) [20:43:54] 10Beta-Cluster-Infrastructure, 10Release Pipeline, 10serviceops, 10Core Platform Team Backlog (Next), and 2 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Ottomata) An example of environmental differences: service-runner uses statsd. In prod we use... [20:45:23] 10Gerrit, 10Operations, 10cloud-services-team, 10serviceops: Change /r/p/ to /r/ on all hosts (where https://gerrit.wikimedia.org/r/p/ exists) - https://phabricator.wikimedia.org/T222093 (10Paladox) [20:46:12] 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Ottomata) GREAT I just made a stretch instance deployment-eventgate-1, and will move eventgate-analytics there. I will also run eventgate-main there in a different c... [20:53:34] 10Gerrit, 10Operations, 10cloud-services-team, 10serviceops: Change /r/p/ to /r/ on all hosts (where https://gerrit.wikimedia.org/r/p/ exists) - https://phabricator.wikimedia.org/T222093 (10Paladox) [20:58:41] (03PS7) 10Dduvall: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) [20:58:43] (03PS7) 10Dduvall: pipeline: Provide a rickety but useful system test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 [20:59:31] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:00:08] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 (owner: 10Dduvall) [21:05:11] (03PS8) 10Dduvall: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) [21:05:13] (03PS8) 10Dduvall: pipeline: Provide a rickety but useful system test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 [21:06:00] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:06:17] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 (owner: 10Dduvall) [21:28:16] (03CR) 10Thcipriani: [C: 03+1] "+1 in case you want to address some nitpicks, effectively a +2 if you don't want to address nitpicks :)" (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:44:55] (03PS9) 10Dduvall: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) [21:44:57] (03PS9) 10Dduvall: pipeline: Provide a rickety but useful system test [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 [21:45:22] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:45:43] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502919 (owner: 10Dduvall) [21:47:34] (03CR) 10Dduvall: pipeline: Builder and stage implementation (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:49:24] (03CR) 10Dduvall: "thcipriani: Addressed your latest comments, and I've moved the setup/teardown constants into PipelineStage (after a second look I think th" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:53:37] (03CR) 10Thcipriani: [C: 03+2] pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:53:59] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:54:01] (03Merged) 10jenkins-bot: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [21:54:34] (03CR) 10jenkins-bot: pipeline: Builder and stage implementation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/502918 (https://phabricator.wikimedia.org/T210267) (owner: 10Dduvall) [23:04:14] 10Project-Admins: Create project WTS2019 - https://phabricator.wikimedia.org/T222929 (10Reedy) [23:04:24] 10Project-Admins: Create project Wiki goes Caribbean - https://phabricator.wikimedia.org/T222930 (10Reedy) [23:44:33] 10Gerrit, 10Wikimedia-General-or-Unknown, 10Documentation, 10Epic, and 4 others: Update Gerrit /r/p/ links to /r/ - https://phabricator.wikimedia.org/T218844 (10Dzahn) rewrite rules have been deployed to prod. we tested cloning still works. This removed a blocker for T200739 [23:44:36] 10MediaWiki-Releasing, 10Release-Engineering-Team (Kanban), 10MW-1.33-release: Branch REL1_33 for MediaWiki and deployed extensions - https://phabricator.wikimedia.org/T220653 (10greg) p:05Triageβ†’03High [23:44:57] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Upgrade to Gerrit 2.16.7 - https://phabricator.wikimedia.org/T200739 (10Dzahn) [23:45:07] 10Gerrit, 10Wikimedia-General-or-Unknown, 10Documentation, 10Epic, and 4 others: Update Gerrit /r/p/ links to /r/ - https://phabricator.wikimedia.org/T218844 (10Dzahn) [23:46:22] 10Deployments, 10Release-Engineering-Team (Backlog), 10HHVM, 10Wikimedia-Incident: Figure out why HHVM kept running stale code for hours - https://phabricator.wikimedia.org/T181833 (10greg) p:05Triageβ†’03Low Given the move to php7, this is low if not lowest priority for now. [23:47:13] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Upgrade to Gerrit 2.16.8 - https://phabricator.wikimedia.org/T200739 (10Paladox) [23:49:07] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Experiment with continuous deployment using Blubberoid - https://phabricator.wikimedia.org/T214158 (10greg) p:05Triageβ†’03Normal [23:49:59] 10Continuous-Integration-Config, 10Release-Engineering-Team (Watching / External), 10Security-Team: Add tests/CI to wikimedia/security/puppet - https://phabricator.wikimedia.org/T217123 (10greg) moving to our watching project until there are tests to run. [23:51:03] 10Release-Engineering-Team (Kanban), 10serviceops, 10Release Pipeline (Blubber): Add k8s credentials for Blubberoid continuous deployment - https://phabricator.wikimedia.org/T217147 (10greg) p:05Triageβ†’03Normal [23:51:22] 10Release-Engineering-Team (Kanban), 10Developer-Advocacy: Tech Talks Proposal 2019: new CI system candidate demonstration - https://phabricator.wikimedia.org/T220695 (10greg) p:05Triageβ†’03Normal [23:54:30] PROBLEM - Citoid on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 1970: Connection refused