[12:15:57] 6Release-Engineering, 6Performance-Team, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1552590 (10BBlack) So, our legit requests for recent versions that we expect all have an obvious Referer header of ours, as in:... [12:56:29] greg-g: sorry I missed the releng meeting yesterday… I remembered the reschedule enough to not show up at the original time but not enough to show up at the new time. [13:07:41] 10Continuous-Integration-Config, 10Wikidata: add CI to Wikidata.org and WikimediaBadges extensions - https://phabricator.wikimedia.org/T107834#1552710 (10Lydia_Pintscher) p:5Triage>3Low [13:07:55] 10Continuous-Integration-Config, 10Wikidata: [Task] Add CI to Wikidata.org and WikimediaBadges extensions - https://phabricator.wikimedia.org/T107834#1552713 (10Lydia_Pintscher) [13:30:35] (03PS2) 10Aude: Add WikimediaBadges and Wikidata.org extensions [integration/config] - 10https://gerrit.wikimedia.org/r/232445 (https://phabricator.wikimedia.org/T107834) [13:35:34] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review: [Task] Add CI to Wikidata.org and WikimediaBadges extensions - https://phabricator.wikimedia.org/T107834#1552797 (10aude) https://gerrit.wikimedia.org/r/#/c/229199/ remains unmerged (and thus not in this weeks deployment branch) because there is... [14:27:52] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1552887 (10GWicke) > Any system that verified that hhvm had correctly restarted would have given @ori a green light too. Correct, but a... [14:34:18] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1552923 (10GWicke) [14:40:59] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review: [Task] Add CI to Wikidata.org and WikimediaBadges extensions - https://phabricator.wikimedia.org/T107834#1552962 (10JanZerebecki) [14:46:09] (03CR) 10JanZerebecki: [C: 032] Add WikimediaBadges and Wikidata.org extensions [integration/config] - 10https://gerrit.wikimedia.org/r/232445 (https://phabricator.wikimedia.org/T107834) (owner: 10Aude) [14:47:27] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-08-18: [Task] Add CI to Wikidata.org and WikimediaBadges extensions - https://phabricator.wikimedia.org/T107834#1552987 (10JanZerebecki) [14:47:42] (03Merged) 10jenkins-bot: Add WikimediaBadges and Wikidata.org extensions [integration/config] - 10https://gerrit.wikimedia.org/r/232445 (https://phabricator.wikimedia.org/T107834) (owner: 10Aude) [14:51:14] !log reload zuul for 700f380..0384ff5 [14:51:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:53:10] 10Deployment-Systems, 6Release-Engineering, 7Epic: Merge to deployed branches instead of a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#1553018 (10Jdforrester-WMF) How are we planning to tag commits and when they go into production if we only have one branch? Is the plan to st... [14:54:23] legoktm: Meh. https://phabricator.wikimedia.org/p/Forrestbot/ – tagging lots of things as wmf20 which actually were wmf19. [14:54:32] * James_F grumps. [15:06:22] (03PS2) 10Florianschmidtwelzow: Remove WikiGrok as dependency from mediawiki-core phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/231579 [15:06:36] (03CR) 10Florianschmidtwelzow: "I'm not sure, if the tests should be removed from jjb, too?" [integration/config] - 10https://gerrit.wikimedia.org/r/231579 (owner: 10Florianschmidtwelzow) [15:19:49] andrewbogott: no worries, we're doing a lot of administrivia-type work (roadmap, etc) [15:24:45] (03CR) 10JanZerebecki: [C: 04-1] "Yes, the occurances from jjb/mediawiki-extensions.yaml and the two other ones from zuul/layout.yaml need to be removed, too. It is easy to" [integration/config] - 10https://gerrit.wikimedia.org/r/231579 (owner: 10Florianschmidtwelzow) [15:34:44] 10Beta-Cluster, 10Traffic, 6operations, 5Patch-For-Review: Upgrade beta-cluster caches to jessie - https://phabricator.wikimedia.org/T98758#1553095 (10demon) 5Open>3Resolved [15:38:35] 6Release-Engineering, 6Performance-Team, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1553139 (10greg) >>! In T102991#1552590, @BBlack wrote: > I took another 10-minute sample, looking only at 1.26wmfN where N is... [15:40:23] 6Release-Engineering, 10MediaWiki-Unit-tests, 5Patch-For-Review: mediawiki: Release tarballs do not contain XCF images for test suite - https://phabricator.wikimedia.org/T103683#1553192 (10demon) 5Open>3Resolved a:3demon [15:45:27] 10Browser-Tests, 10Wikidata, 5Patch-For-Review, 3Wikidata-Sprint-2015-06-30, 3Wikidata-Sprint-2015-08-11: [Task] Adjust browsertests for references - https://phabricator.wikimedia.org/T92249#1553245 (10Lydia_Pintscher) [15:46:39] 10Continuous-Integration-Infrastructure: make-release doesn't include the correct extension branch - https://phabricator.wikimedia.org/T59088#1553246 (10demon) Wasn't this working ok @csteipp when we did the release the other day? [15:53:12] 10Continuous-Integration-Infrastructure: make-release doesn't include the correct extension branch - https://phabricator.wikimedia.org/T59088#1553266 (10csteipp) 5Open>3Resolved a:3csteipp Yep! [16:01:15] 6Release-Engineering, 6Performance-Team, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1553321 (10BBlack) Honestly, I have no idea, but probably something like that... [16:03:43] 6Release-Engineering, 6Performance-Team, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1553328 (10BBlack) Actually, it can't be as simple as FB caching and serving outdated text to a browser, which then fetches the... [16:09:46] 6Release-Engineering, 6Performance-Team, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1553482 (10BBlack) some details from others' investigations: https://www.webmasterworld.com/search_engine_spiders/4436904.htm [16:10:55] (03CR) 10Dduvall: [C: 032] Add jenkins test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 (owner: 10Paladox) [16:12:35] (03PS18) 10Dduvall: Add jenkins test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 (owner: 10Paladox) [16:14:40] (03CR) 10Dduvall: [C: 032] Add jenkins test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 (owner: 10Paladox) [16:16:00] 6Release-Engineering, 10ReleaseTaggerBot, 5Patch-For-Review: ForrestBot not working since 2015-08-14? - https://phabricator.wikimedia.org/T109519#1553489 (10Jdforrester-WMF) 5Resolved>3Open We probably should fix all the mis-attributed wmf20 ones before closing this. [16:16:15] (03Merged) 10jenkins-bot: Add jenkins test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 (owner: 10Paladox) [16:17:12] !log Reloading Zuul to deploy I48ab39e330ebc71266b72cae8449cc2f6da495fe [16:17:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:17:53] (03PS13) 10Dduvall: Add jenkins tests for EditUser [integration/config] - 10https://gerrit.wikimedia.org/r/228500 (owner: 10Paladox) [16:19:41] (03CR) 10Dduvall: [C: 032] Add jenkins tests for EditUser [integration/config] - 10https://gerrit.wikimedia.org/r/228500 (owner: 10Paladox) [16:20:49] (03PS3) 10Florianschmidtwelzow: Remove WikiGrok as dependency from mediawiki-core phpunit [integration/config] - 10https://gerrit.wikimedia.org/r/231579 [16:21:16] (03Merged) 10jenkins-bot: Add jenkins tests for EditUser [integration/config] - 10https://gerrit.wikimedia.org/r/228500 (owner: 10Paladox) [16:21:42] !log Reloading Zuul to deploy I6815cd66169ee8f6fbb5ea394e3a10ce6b6e7609 [16:21:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:22:18] Is jenkins broken? [16:22:31] https://integration.wikimedia.org/ci/job/mwext-testextension-zend/6761/console [16:22:35] 16:19:07 ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) [16:24:17] Krenair: repeating from yesterday morning related commentary: [16:24:19] 16:32 < jzerebeck> !log offlined integration-slave-precise-1013 : Fails to connect to mysl. /etc/init.d/mysql start fails. [16:24:23] 16:32 < qa-morebo> Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:24:26] 16:33 < stan!~stan@2001:41c1:4:8000:992d:719:b10f:3644 [Ping timeout: 246 seconds] [16:24:29] 16:45 <+ greg-g> jzerebecki: is there a task for that? [16:24:31] 16:46 <+ greg-g> that == failing to connect to mysql on 1013 [16:24:34] 16:50 < CFisch_WMDE!chfi_wmde@nat/wmf/x-zijdhksuxtiyrwrf [Quit: Leaving.] [16:24:37] 17:00 <+thciprian> jzerebecki: I had this same problem a few weeks ago with a different slave, oddly the problem ended up being: /var/run/elasticsearch wasn't a directory, so elasticsearch hung on starting so mysql table cleanup and setup never ran. The give away was 20 `/etc/init.d/elasticsearch start` hanging when looking at `ps aux | grep -i elastic` [16:24:42] 17:02 <+thciprian> basically puppet wouldn't run, so mysql didn't work [16:25:30] This was rechecked, what's the chance of it hitting the same bad server twice in a row? [16:25:35] Or are multiple servers broken? [16:26:29] that was 1013, the job you linked was 1014 [16:26:45] I don't have time to investigate, and I'll be taking tyler into a 1:1 with me in 4 minutes [16:27:04] marxarelli: ^ when you have a chance [16:27:10] yep yep [16:29:34] !log Investigating stopped mysql on integration-slave-precise-1014 [16:29:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:32:53] !log Running apt-get install --reinstall elasticsearch to re-create missing /var/run/elasticsearch (and possibly others) directory [16:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:35:07] !log Verified that elasticsearch and mysql are now running after the reinstall and a manual puppet run [16:35:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:35:35] Krenair, greg-g: should be good now [16:35:45] i'll check the other precise machines [16:39:20] yep [16:39:22] ty [16:40:07] 6Release-Engineering, 6Performance-Team, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1553544 (10BBlack) My thoughts here at this point are: 1) We should delete the branches automatically and regularly, with some... [16:43:17] np [16:58:01] uh oh [16:58:15] wait, bd808, were you just on tyler and I's 1:1 call? we made references to skynet [16:58:49] If you can hear this you are part of the resistance [16:58:54] * greg-g now sees where the T1000 reference was made [16:59:11] surprised that wasn't a registered nick [16:59:24] it was [16:59:42] but that doesn't actually stop you from using it for a few seconds [16:59:48] so you can auth [17:00:05] oh right [17:00:12] irc !== secure [17:03:44] isn't there a setting to make that more aggressively protective? [17:08:09] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-SpamBlacklist, 5Patch-For-Review: Figure out a system to override default settings when in test context - https://phabricator.wikimedia.org/T89096#1553749 (10dduvall) >>! In T89096#1552164, @Legoktm wrote: > When people are running tests locally... [17:09:17] chasemp: maybe? I think you can set a hostmask for protection too [17:18:26] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-SpamBlacklist, 5Patch-For-Review: Figure out a system to override default settings when in test context - https://phabricator.wikimedia.org/T89096#1553780 (10Jdlrobson) We already have this situation. We have a README that explains certain confi... [17:20:46] 6Release-Engineering, 6Performance-Team, 10Traffic, 6operations, 7Varnish: Verify traffic to static resources from past branches does indeed drain - https://phabricator.wikimedia.org/T102991#1553790 (10Nemo_bis) [17:27:45] sidenote: I like my team, that is all. [17:32:48] (03CR) 10Paladox: "Thanks." [integration/config] - 10https://gerrit.wikimedia.org/r/228500 (owner: 10Paladox) [17:36:31] (03PS6) 10Paladox: Change jslint to jshint in SkelJS [integration/config] - 10https://gerrit.wikimedia.org/r/228670 [17:37:03] (03PS4) 10Paladox: Add jenkings test for SpamDiffTool [integration/config] - 10https://gerrit.wikimedia.org/r/228622 [17:38:26] (03Abandoned) 10Paladox: Remove beta job for math extension [integration/config] - 10https://gerrit.wikimedia.org/r/228263 (owner: 10Paladox) [17:38:34] (03PS4) 10Paladox: Update BreadCrumbs tests [integration/config] - 10https://gerrit.wikimedia.org/r/228136 [17:38:55] (03PS10) 10Paladox: Update HitCounters tests [integration/config] - 10https://gerrit.wikimedia.org/r/227438 [17:40:34] (03PS9) 10Paladox: Update Blueprint tests [integration/config] - 10https://gerrit.wikimedia.org/r/226542 [17:41:45] (03PS15) 10Paladox: Update interwiki tests [integration/config] - 10https://gerrit.wikimedia.org/r/225313 [17:42:40] (03PS7) 10Paladox: Add check for json in TwitterLogin [integration/config] - 10https://gerrit.wikimedia.org/r/225711 [17:48:10] (03PS4) 10Paladox: Update EditSimilar tests [integration/config] - 10https://gerrit.wikimedia.org/r/228641 [17:49:06] (03Abandoned) 10Paladox: Fix tests by adding MobileFrontend as dependency to TimedMediaHandler [integration/config] - 10https://gerrit.wikimedia.org/r/228803 (owner: 10Paladox) [18:36:46] 6Release-Engineering, 5Testing Initiative 2015: Guides for initializing a test suite: unit testing & browser testing - https://phabricator.wikimedia.org/T108107#1554174 (10dduvall) a:5dduvall>3None [18:37:04] greg-g, Krenair, marxarelli: that is https://phabricator.wikimedia.org/T109497 [18:40:19] (03CR) 10MarkAHershberger: [C: 031] "ping, hashar, could you +2" [integration/config] - 10https://gerrit.wikimedia.org/r/227438 (owner: 10Paladox) [18:41:32] (03CR) 10Greg Grossmeier: "In the grand future of Differential you'd be able to see that hashar has a red dot next to his name indicating he's out (on vacation). He'" [integration/config] - 10https://gerrit.wikimedia.org/r/227438 (owner: 10Paladox) [18:41:39] 10Continuous-Integration-Infrastructure, 5MW-1.26-release, 5Patch-For-Review: Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches - https://phabricator.wikimedia.org/T90303#1554206 (10JanZerebecki) This currently does not work, because the jobs using composer right now... [18:43:01] (03PS1) 10JanZerebecki: [WIP] Change target for clone to src. [integration/config] - 10https://gerrit.wikimedia.org/r/232558 (https://phabricator.wikimedia.org/T90303) [19:10:33] 6Release-Engineering, 10ReleaseTaggerBot, 5Patch-For-Review: ForrestBot not working since 2015-08-14? - https://phabricator.wikimedia.org/T109519#1554314 (10mmodell) >>! In T109519#1551843, @Legoktm wrote: > Sorry, I misnamed it in my head. > > So we're planning to get rid of branches? Is this documented so... [19:42:11] 10Deployment-Systems, 6Release-Engineering, 7Epic: Merge to deployed branches instead of a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#1554422 (10mmodell) >>! In T89945#1553018, @Jdforrester-WMF wrote: > How are we planning to tag commits and when they go into production if w... [20:40:55] 10Continuous-Integration-Config, 6Performance-Team, 7JavaScript: Run PhantomJS across set of Wikimedia wiki pages to ensure sane JavaScript - https://phabricator.wikimedia.org/T71519#717377 (10Krenair) [21:59:41] Yippee, build fixed! [21:59:45] Project beta-code-update-eqiad build #68829: FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/68829/ [22:03:13] we probably should have marked the slaves offline in jenkins before the migration started [22:07:47] 6RelEng-Admin, 15User-greg: Admin: fill out Summer Managers Offsite - Roadmap 2015-2016 spreadsheet - https://phabricator.wikimedia.org/T108984#1555050 (10greg) 5Open>3Resolved [22:22:59] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1555130 (10Tgr) >>! In T102566#1548300, @Tau wrote: > Can this issue be fixed if I upgrade from 1.23.3 to 1.23.10?... [22:46:39] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1555192 (10BBlack) @Tgr Do we have any immediate plans to fix those anyways, or a sane plan to fix them that would a... [22:49:31] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1555215 (10BBlack) Answering my own question: there's mw/core patches attached to both, last activity about a month... [23:08:20] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1555371 (10Tgr) Nothing apart from lack of reviewers, I think. I can review them in the next couple days, but that o... [23:15:16] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1555390 (10Tgr) To answer myself, we log them in `api-feature-usage` but I have no idea how to tell curl vs. fopen f... [23:23:54] 6RelEng-Admin: Setup meeting with $people to discuss code hosting exception policy - https://phabricator.wikimedia.org/T109657#1555438 (10greg) 3NEW a:3greg [23:24:22] 6RelEng-Admin, 15User-greg: Admin: fill out Summer Managers Offsite - Roadmap 2015-2016 spreadsheet - https://phabricator.wikimedia.org/T108984#1555451 (10greg) 5Resolved>3Open [23:31:39] 10Beta-Cluster, 7Pywikibot-tests: paraminfo for contenttranslationlist is broken on beta wikis - https://phabricator.wikimedia.org/T109659#1555485 (10jayvdb) 3NEW [23:32:02] 10Beta-Cluster, 10ContentTranslation-Deployments, 7Pywikibot-tests: paraminfo for contenttranslationlist is broken on beta wikis - https://phabricator.wikimedia.org/T109659#1555492 (10jayvdb) [23:32:24] (03PS1) 10Dduvall: Decouple remote browser from Sauce Cucumber formatter [selenium] - 10https://gerrit.wikimedia.org/r/232660 [23:33:32] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 7Pywikibot-tests: paraminfo for contenttranslationlist is broken on beta wikis - https://phabricator.wikimedia.org/T109659#1555485 (10jayvdb) [23:34:48] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10MediaWiki-API, 7Pywikibot-tests: prevent modules with broken paraminfo being deployed to production wikis - https://phabricator.wikimedia.org/T108322#1555507 (10jayvdb) It has happened again: {T109659} [23:36:10] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 7Pywikibot-tests: paraminfo for contenttranslationlist is broken on beta wikis - https://phabricator.wikimedia.org/T109659#1555528 (10Legoktm) Aug 19 23:32:15 deployment-mediawiki02: #012Fatal error: File not foun... [23:41:14] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 7Pywikibot-tests: paraminfo for contenttranslationlist is broken on beta wikis - https://phabricator.wikimedia.org/T109659#1555558 (10jayvdb) [23:42:34] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 7Pywikibot-tests: paraminfo for contenttranslationlist is broken on beta wikis - https://phabricator.wikimedia.org/T109659#1555566 (10jayvdb) [23:51:22] 10Beta-Cluster, 10ContentTranslation-Deployments, 10MediaWiki-extensions-ContentTranslation, 7Pywikibot-tests: paraminfo for contenttranslationlist is broken on beta wikis - https://phabricator.wikimedia.org/T109659#1555598 (10jayvdb) [23:53:03] (03PS2) 10Legoktm: [WIP] Add job to run pywikibot tests against beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/220384 (https://phabricator.wikimedia.org/T100903) [23:55:16] (03PS3) 10Legoktm: [WIP] Add job to run pywikibot tests against beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/220384 (https://phabricator.wikimedia.org/T100903) [23:55:44] (03CR) 10John Vandenberg: [WIP] Add job to run pywikibot tests against beta cluster (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/220384 (https://phabricator.wikimedia.org/T100903) (owner: 10Legoktm) [23:57:48] (03PS1) 10Dduvall: Reorganize to separate out Cucumber support [selenium] - 10https://gerrit.wikimedia.org/r/232663