[12:28:41] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging: deployment-kafka01 / partition is full - https://phabricator.wikimedia.org/T168564#3370226 (10hashar) Merci! [13:24:46] 10Gerrit, 10Release-Engineering-Team, 10Discovery-Analysis, 10Patch-For-Review: Cannot push to Gerrit repo without review - https://phabricator.wikimedia.org/T168588#3370343 (10hashar) If I am not mistaken, Gerrit accepts reviews for merge commits. And assuming the changes in the develop branch have been m... [13:31:04] 10Gerrit, 10Release-Engineering-Team, 10Discovery-Analysis, 10Patch-For-Review: Cannot push to Gerrit repo without review - https://phabricator.wikimedia.org/T168588#3370352 (10Paladox) @hashar I think they want to do git push origin HEAD:refs/heads/master [13:35:08] !log Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group . That also makes him an owner to wikimedia/discovery/* [13:35:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:35:14] !log Gerrit: adding Bearloga (Mikhail Popov) to the 'search' group . That also makes him an owner to wikimedia/discovery/* - T168588 [13:35:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:35:17] T168588: Cannot push to Gerrit repo without review - https://phabricator.wikimedia.org/T168588 [13:38:23] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Discovery-Analysis, 10Patch-For-Review: Cannot push to Gerrit repo without review - https://phabricator.wikimedia.org/T168588#3370368 (10hashar) a:03hashar @Paladox proposed a change to the [[ https://gerrit.wikimedia.org/r/#/admin/groups/251,members | Ger... [14:02:01] 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase2), 10Patch-For-Review: Deploy logstash/plugins with scap3 - https://phabricator.wikimedia.org/T165748#3370432 (10thcipriani) >>! In T165748#3369771, @Gehel wrote: > @thcipriani Thanks for all this! I can be available to deploy this today 6pm C... [14:10:57] RECOVERY - Puppet staleness on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [3600.0] [14:11:34] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3370537 (10hashar) [14:11:37] 10Deployment-Systems, 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase1), 10scap2, and 2 others: Deploy jobrunner with scap3 (Trebuchet jobrunner/jobrunner) - https://phabricator.wikimedia.org/T129148#3370533 (10hashar) 05Resolved>03Open So that is not fully **done**. We need to restar... [14:13:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Citoid, 10VisualEditor, and 2 others: Beta cluster varnish fails VCL compilation because citoid.wmflabs.org does not resolve - https://phabricator.wikimedia.org/T168519#3370558 (10hashar) 05Open>03Resolved a:03hashar What I suspect... [14:22:46] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Discovery-Analysis, 10Patch-For-Review: Cannot push to Gerrit repo without review - https://phabricator.wikimedia.org/T168588#3370646 (10hashar) p:05Triage>03Normal [14:31:43] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Upstream, 10WorkType-NewFunctionality: JJB should support YAML axis - https://phabricator.wikimedia.org/T128462#3370664 (10hashar) [14:33:15] 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10Documentation, 10Ruby, and 2 others: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#3370672 (10zeljkofilipin) Looks like this is already resolved in 43e3a40a68e9e79868008d4276df90f199d44ff9 by @hashar. [14:33:41] Yippee, build fixed! [14:33:42] Project selenium-WikiLove ยป firefox,beta,Linux,BrowserTests build #432: 09FIXED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/432/ [14:36:40] 10Release-Engineering-Team (Kanban), 10MediaWiki-Vagrant, 10Documentation, 10Patch-For-Review, and 3 others: Document RSpec workflow on MediaWiki-Vagrant - https://phabricator.wikimedia.org/T97464#3370695 (10zeljkofilipin) I have updated the docs in the above patch. It is no longer needed to use bundler <... [15:06:59] 10Release-Engineering-Team, 10Jenkins: Upgrade jenkins to 2.60.1 (new lts release) - https://phabricator.wikimedia.org/T168644#3370797 (10Paladox) [15:07:28] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Upstream, 10WorkType-NewFunctionality: Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms - https://phabricator.wikimedia.org/T103351#1387714 (10Paladox) [15:07:30] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure: Reenable ssh MAC/KEX hardening on beta cluster and integration labs project - https://phabricator.wikimedia.org/T100518#3370815 (10Paladox) [15:07:32] 10Release-Engineering-Team, 10Jenkins: Upgrade jenkins to 2.60.1 (new lts release) - https://phabricator.wikimedia.org/T168644#3370813 (10Paladox) [15:07:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Jenkins, 10Patch-For-Review: Upgrade jenkins server and jenkins slaves to java 8 - https://phabricator.wikimedia.org/T162828#3176110 (10Paladox) [15:07:53] 10Release-Engineering-Team, 10Jenkins: Upgrade jenkins to 2.60.1 (new lts release) - https://phabricator.wikimedia.org/T168644#3370797 (10Paladox) [15:10:57] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Upstream, 10WorkType-NewFunctionality: JJB should support YAML axis - https://phabricator.wikimedia.org/T128462#3370840 (10hashar) [15:11:17] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3370841 (10zeljkofilipin) >>! In T164721#3368608, @Jdlrobson wrote: > I personally don't use Vagrant. Ok, th... [15:11:26] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Upstream, 10WorkType-NewFunctionality: JJB should support YAML axis - https://phabricator.wikimedia.org/T128462#2075587 (10hashar) I have addressed the few comments that were pending in OpenStack Gerrit and rebased the serie of patches. [15:22:19] thcipriani: I sent the scap config patch to restart jobrunner AND jobchron https://gerrit.wikimedia.org/r/#/c/360856/1/scap/scap.cfg [15:22:27] which made me realize that ther eis a service_port: 9005 [15:22:31] so no clue how that will catch that [15:22:44] I am all confused :] [15:22:52] most probably that will do the right thing [15:23:41] hashar: so what will happen is both services will be restarted and then scap will check that port 9005 is accepting tcp connections [15:24:17] the jobchron thing is obviously new, but the jobrunner port 9005 is just a straight port over from the trebuchet setup [15:24:41] adding additional ports to check is a different refactor I think :) [15:24:51] :D [15:25:08] 9005 is just the HHVM rpc entry point apparently [15:25:31] ah [15:25:57] yeah that is Apache2 (confirmed on deployment-jobrunner02) [15:26:06] so that will work [15:26:29] I have reopened the task we had and will follow up whenever scap 3.6 is released to prodo [15:27:22] ok, I'll wrangle up whatever we want in the new release today and try to get it out shortly [15:27:59] \o/ [15:28:20] currently for scap 3.6, there is pretty much only the services change and a change to docs, but there are some other patches that are probably ready. [15:28:56] there is probably no hurries on the jobrunner front [15:29:29] sure, would just be nice to check it off the list. Deploying logstash/plugins in 30 minutes or so, so that's one less thing. [15:32:21] hashar: I think that a lot of the kinks with the operations/puppet docker thing seem to be worked out https://integration.wikimedia.org/ci/job/operations-puppet-tests-docker/buildTimeTrend [15:33:22] (i.e., seems to fail and abort only when it's supposed to, seems to match the -jessie job) [15:40:40] thcipriani: neat! will try to remember to have a look at it tomorrow [15:41:01] awesome, thanks :) [15:41:33] and find out why it is faster!!! :D [15:42:57] I am off. [16:34:57] 10Continuous-Integration-Config: Reject non-executable files with execute bits with a build check - https://phabricator.wikimedia.org/T168659#3371176 (10Umherirrender) [16:51:38] (03PS1) 10Phedenskog: Publish WebPageTest job status to wikimedia-perf-bots [integration/config] - 10https://gerrit.wikimedia.org/r/360875 [17:24:39] 10MediaWiki-Codesniffer: Sniff to disallow PHP 7 Unicode escape syntax - https://phabricator.wikimedia.org/T168669#3371426 (10Legoktm) [17:26:13] 10Release-Engineering-Team, 10Jenkins: Upgrade jenkins to 2.60.1 (new lts release) - https://phabricator.wikimedia.org/T168644#3371440 (10Paladox) [17:29:17] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3371453 (10thcipriani) [17:29:19] 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase2), 10Patch-For-Review: Deploy logstash/plugins with scap3 - https://phabricator.wikimedia.org/T165748#3371451 (10thcipriani) 05Open>03Resolved All done! @Gehel merged and did the deploy today: https://twitter.com/wikimediatech/status/87794... [17:29:24] RainbowSprinkles according to luca here https://groups.google.com/forum/#!topic/repo-discuss/_5iJcIsIa2Y we may not have done the reindex properly (ie offline index, that's probaly why it was so fast when we upgraded from 2.12 to 2.13.) [17:29:37] Gerrit can start even without the index for accounts. [17:32:02] that would have explaned it. [17:32:11] Another user did a full reindex which fixed it for them. [17:34:30] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3371466 (10Paladox) Luca found the leading cause to this. It's because we did not do the full reindex properly which did not... [17:37:31] 10Release-Engineering-Team (Kanban), 10Scap (Scap3-Adoption-Phase2), 10Patch-For-Review: Deploy logstash/plugins with scap3 - https://phabricator.wikimedia.org/T165748#3371479 (10Gehel) I did not know we have a twitter feed ... [17:37:43] 10Gerrit, 10Release-Engineering-Team, 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3371480 (10Paladox) [17:52:49] 10Gerrit, 10Release-Engineering-Team, 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3371559 (10Paladox) p:05Triage>03High Setting high as this needs to be done to fix T152640 and prevent it returning in any future release. [17:55:47] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Discovery-Analysis, 10Patch-For-Review: Cannot push to Gerrit repo without review - https://phabricator.wikimedia.org/T168588#3371571 (10mpopov) 05Open>03Resolved I followed @hashar's instructions (I didn't even know about the access GUI!) and everything... [18:13:50] (03PS2) 10Phedenskog: Publish WebPageTest job status to wikimedia-perf-bots [integration/config] - 10https://gerrit.wikimedia.org/r/360875 (https://phabricator.wikimedia.org/T126216) [18:23:45] Project beta-scap-eqiad build #160837: 04FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160837/ [18:26:53] Yippee, build fixed! [18:26:53] Project beta-scap-eqiad build #160838: 09FIXED in 2 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160838/ [19:02:18] !log cherry-picking gerrit:360891/1 (T163922) [19:02:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:02:22] T163922: Create a URL rewrite to handle the /data/ path for canonical URLs for machine readable page content - https://phabricator.wikimedia.org/T163922 [19:04:41] 10Scap, 10Discovery, 10Interactive-Sprint, 10Maps (Kartotherian), 10Patch-For-Review: Break Kartotherian scap3 deployment into 2 groups - https://phabricator.wikimedia.org/T147337#3371775 (10debt) 05Open>03Resolved [19:19:55] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Wikimedia-Incident: Disallow blocked users on mediawiki to create accounts on phabricator - https://phabricator.wikimedia.org/T162996#3371862 (10greg) [19:49:51] 10Release-Engineering-Team (Kanban), 10MW-1.30-release-notes (WMF-deploy-2017-06-27_(1.30.0-wmf.7)), 10Patch-For-Review, 10Release, 10Train Deployments: MW-1.30.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T167535#3371951 (10mmodell) [20:49:36] (03PS1) 10Legoktm: Include sniff warning/error codes in test output [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/360938 [20:55:57] Project beta-scap-eqiad build #160855: 04FAILURE in 2 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160855/ [21:02:36] something is amiss with openstack that will probably delay CI tests until I sort it out. [21:05:48] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Fundraising Sprint Nitpicking, and 6 others: Continuous integration: DonationInterface needs composer variant - https://phabricator.wikimedia.org/T141309#3372153 (10DStrine) a:05awight>03None [21:08:17] Yippee, build fixed! [21:08:18] Project beta-scap-eqiad build #160856: 09FIXED in 4 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160856/ [21:12:54] ok, the openstack/CI issue should be fixed [21:18:34] 10Continuous-Integration-Infrastructure: mwext-donationinterfacecore-REL1_27-testextension-zend55 always fails with Can't connect to local MySQL server - https://phabricator.wikimedia.org/T168687#3372295 (10Umherirrender) [21:26:04] 10Continuous-Integration-Infrastructure: mwext-donationinterfacecore-REL1_27-testextension-zend55 always fails with Can't connect to local MySQL server - https://phabricator.wikimedia.org/T168687#3372295 (10Paladox) mysql needs restarting on there. @hashar would you be able to restart mysql on there please? [21:31:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): mwext-donationinterfacecore-REL1_27-testextension-zend55 always fails with Can't connect to local MySQL server - https://phabricator.wikimedia.org/T168687#3372337 (10hashar) 05Open>03Resolved a:03hashar Indeed: ``` root@integra... [21:32:17] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: CiviCRM: lint json and php files using composer - https://phabricator.wikimedia.org/T163781#3372349 (10mmodell) [21:33:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): mwext-donationinterfacecore-REL1_27-testextension-zend55 always fails with Can't connect to local MySQL server - https://phabricator.wikimedia.org/T168687#3372378 (10Paladox) Thanks :) [21:37:07] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10Patch-For-Review, 10WorkType-Maintenance: Switch wikimedia/fundraising/slander to use tox as an entry point - https://phabricator.wikimedia.org/T114250#3372557 (10mmodell) [21:42:17] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3372854 (10demon) Ok, I guess we can reindex....again. [21:43:02] 10Gerrit, 10Release-Engineering-Team, 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3372915 (10demon) I don't need a date, reindexing accounts will take all of thirty seconds. [21:43:38] Project beta-code-update-eqiad build #160954: 04FAILURE in 37 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/160954/ [21:43:56] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10WorkType-Maintenance: wikimedia/fundraising/tools CI jobs are broken - https://phabricator.wikimedia.org/T117818#3372984 (10mmodell) [21:44:21] 10Beta-Cluster-Infrastructure, 10Analytics, 10Analytics-EventLogging, 10Fundraising-Backlog, and 4 others: Beta Cluster EventLogging data is disappearing? - https://phabricator.wikimedia.org/T112926#3373008 (10mmodell) [21:44:27] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising Tech Backlog, 10Fundraising-Backlog, and 2 others: Enable PHPUnit testing on the wikimedia/fundraising/SmashPig repo - https://phabricator.wikimedia.org/T104264#3373020 (10mmodell) [21:44:48] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Support environment variables in configuration - https://phabricator.wikimedia.org/T168425#3373054 (10dduvall) [21:44:58] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Kanban), 10FR-Smashpig, 10Fundraising-Backlog, and 3 others: Disable fundraising CI jobs that are non-voting and always fail - https://phabricator.wikimedia.org/T160476#3373056 (10mmodell) [21:46:21] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3373154 (10demon) 05Open>03Resolved a:03demon ``` gerrit2@cobalt /var/lib/gerrit2/review_site$ java -jar bin/gerrit.war reindex --index acc... [21:53:36] Yippee, build fixed! [21:53:37] Project beta-code-update-eqiad build #160955: 09FIXED in 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/160955/ [21:55:13] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Bad empty CI jobs on wikimedia/fundraising/crm deployment branch - https://phabricator.wikimedia.org/T120881#3373502 (10mmodell) [21:55:22] 10Release-Engineering-Team (Kanban), 10Fundraising-Backlog, 10MediaWiki-Vagrant, 10Wikimedia-Fundraising, and 3 others: Vagrant Fundraising role needs to be able to run a specific MediaWiki branch - https://phabricator.wikimedia.org/T78739#3373506 (10mmodell) [21:57:52] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3373515 (10Paladox) @demon i meant a full index, including changes. But i guess that works :). thanks. [21:59:53] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3373520 (10demon) Why would the changes need to be reindexed if we're talking about accounts? This whole thing is stupid mess.... [22:01:00] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3373523 (10demon) Plus, I disagree with the assertion that we didn't do a full reindex. We did. Twice. [22:01:51] (03PS1) 10Legoktm: Add sniff to prevent against using PHP 7's Unicode escape syntax [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/360987 (https://phabricator.wikimedia.org/T168669) [22:02:00] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Operations: Setup maintenance date to reindex gerrit (offline reindex) - https://phabricator.wikimedia.org/T168670#3373525 (10Paladox) >>! In T168670#3373523, @demon wrote: > Plus, I disagree with the assertion that we didn't do a full reindex. We did. Twice.... [22:05:25] Project beta-scap-eqiad build #160861: 04FAILURE in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160861/ [22:11:51] (03CR) 10Chad: "Bahahahaha, from the RFC: "A further use is to produce characters you can't type on your keyboard. If you are unable to type the emoji for" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/360987 (https://phabricator.wikimedia.org/T168669) (owner: 10Legoktm) [22:15:34] Yippee, build fixed! [22:15:34] Project beta-scap-eqiad build #160862: 09FIXED in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/160862/ [22:26:09] 10Gerrit, 10Analytics-Tech-community-metrics, 10Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3373558 (10Paladox) The change on GitHub is https://github.com/wikimedia/operations-debs-kafka/commit/dbed4e47a6df5028263d62eb6ec97daa588... [23:03:59] Hmmm got a beta cluster error: https://meta.wikimedia.beta.wmflabs.org/wiki/Special:Notifications [23:12:09] AndyRussG: wfm [23:13:51] greg-g: I also get an error in the pop-up for alerts--says, "Failed to fetch notifications [23:14:08] 10Release-Engineering-Team (Kanban), 10Labs, 10Phabricator, 10wikitech.wikimedia.org, and 2 others: Blocking an account on wikitech should disable LDAP logins - https://phabricator.wikimedia.org/T168692#3373658 (10mmodell) [23:14:33] AndyRussG: all of that page works for me, and I had an unread notification :/ [23:17:35] greg-g: Hmmm... I should check logstash [23:17:53] Maybe I have an unread evil notification [23:23:59] (03PS1) 10Jdlrobson: Setup browser test job for Minerva skin [integration/config] - 10https://gerrit.wikimedia.org/r/361012 (https://phabricator.wikimedia.org/T166750) [23:35:28] (03CR) 10Esanders: [C: 031] "Nice" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/360987 (https://phabricator.wikimedia.org/T168669) (owner: 10Legoktm)