[00:00:46] (03CR) 10Paladox: [C: 031] Use php55 jobs for MediaWiki master [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [00:04:38] 10Continuous-Integration-Config: translatewiki.net phplint job should allow PHP 5.4 syntax - https://phabricator.wikimedia.org/T97889#1988242 (10Paladox) Should we use both the php 5.3 and 5.5 template or just the 5.5 template. [00:11:19] 10Continuous-Integration-Config: Create generic mwext-jsduck-publish job and use it - https://phabricator.wikimedia.org/T125473#1988262 (10Legoktm) 3NEW [00:26:16] MarkTraceur: I wasn't getting it locally, ran git-update, and am certainly getting it now, definitely a very recent change. [00:26:26] Yeah [00:30:25] greg-g: https://phabricator.wikimedia.org/T125468 should probably block any deployments of current master to the cluster, unless I'm wrong about it not being fixed [00:39:32] !log precise-1011 slave hasn't had a puppet run in 6 days [00:39:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:40:26] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011.eqiad.wmflabs has not run puppet since Jan 25 - https://phabricator.wikimedia.org/T125474#1988462 (10Legoktm) 3NEW [00:42:43] !log marked integration-slave-precise-1011 as offline [00:42:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:42:50] !log due to T125474 [00:42:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [00:43:48] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1988477 (10Jdforrester-WMF) 3NEW [00:43:58] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1988484 (10Jdforrester-WMF) [00:51:06] James_F: thanks for making that, sadly [00:52:01] MarkTraceur: so... UBN instead of High, then, yes? [00:52:27] greg-g: I mean, high for me, probably UBN if you want wmf12 to go out tomorrow [00:52:38] I personally have no clue how to fix it [00:52:58] MarkTraceur: well, anything that is blocking deployment is an UBN for the people who are responsible [00:52:59] I have tried to get a stacktrace to no effect, tried to untangle the page/.*.php code to no avail [00:53:21] I have to guess that Krenair knows what's happening as the person who touched the code most recently [00:53:22] greg-g: Yeah. :-(' [00:56:32] fwiw, reverting to before https://gerrit.wikimedia.org/r/#/c/267430/2 merged seems to fix it locally [00:56:43] that seems to be related to the patch that was reverted... [00:58:00] Irony [00:58:19] But there was more...sec [00:58:39] thcipriani: No, https://gerrit.wikimedia.org/r/#/c/267433/ was reverted [00:58:42] Oh, related [00:58:45] * MarkTraceur reads [00:59:22] yeah, actually: git revert 6051fcd38374db7e2d984ec535da264cb48d291f locally seems to work... [01:01:14] Hm. [01:02:10] 5Release-Engineering-Epics, 7Epic: [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#1988555 (10Jdlrobson) [01:04:01] OK, so it looks like the trouble is that ImagePage overrides mPage, and still relies on __call to forward to an instance of WikiFilePage. [01:04:13] But it is 19:00 here so I say revert [01:05:07] MarkTraceur: https://gerrit.wikimedia.org/r/#/c/267817/ [01:05:58] OK, testing quick [01:08:00] Merging [01:08:30] kk, will watch beta (as it slowly beta-scap-eqiad s) [01:09:29] So here's a fun question [01:09:43] Why did no unit tests catch that file pages all broke [01:09:53] He says, fully realizing it's probably his fault. [01:10:09] MarkTraceur: Not necessarily your fault, but definitely Your Problem™. [01:10:16] Yeahhh [01:11:29] MarkTraceur, I did a simple revert of that obviously broken commit [01:11:46] Krenair: We had to revert the parent too [01:11:52] ah [01:25:24] 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1988593 (10EBernhardson) 3NEW [01:28:31] 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1988610 (10bd808) Multiversion uses wikimedia/cdb to work around the known bustedness of HHVM and the `dba_*` functions. It probably wouldn't be too hard to re... [01:32:26] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1988614 (10thcipriani) [01:44:59] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1988640 (10Jdforrester-WMF) Tentatively closable, then? [02:01:21] 10Deployment-Systems, 6Performance-Team, 10Traffic, 6operations, 5Patch-For-Review: Varnish cache for /static/$wmfbranch/ doesn't expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#1988662 (10Krinkle) [02:01:41] 10Deployment-Systems, 6Performance-Team, 10Traffic, 6operations, 5Patch-For-Review: Make Varnish cache for /static/$wmfbranch/ expire when resources change within branch lifetime - https://phabricator.wikimedia.org/T99096#1285315 (10Krinkle) [02:07:27] legoktm: So… are we (you) landing https://gerrit.wikimedia.org/r/#/c/266941/ then? [03:15:26] James_F|Away: yeah, just didn't have a good time to sit down and do it [03:19:41] (03CR) 10Legoktm: [C: 032] Use php55 jobs for MediaWiki master [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [03:21:48] (03CR) 10jenkins-bot: [V: 04-1] Use php55 jobs for MediaWiki master [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [03:22:23] kasjdhfkdhf [03:24:18] (03PS10) 10Legoktm: Use php55 jobs for MediaWiki master [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [03:24:47] (03CR) 10Legoktm: [C: 032] Use php55 jobs for MediaWiki master [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [03:27:22] (03Merged) 10jenkins-bot: Use php55 jobs for MediaWiki master [integration/config] - 10https://gerrit.wikimedia.org/r/266941 (https://phabricator.wikimedia.org/T75901) (owner: 10Jforrester) [03:28:57] ready set [03:29:03] !log deploying https://gerrit.wikimedia.org/r/266941 [03:29:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:33:08] https://integration.wikimedia.org/ci/job/mediawiki-core-php55lint/1/console trusty took 2 minutes to clone core [03:43:06] (03PS1) 10Legoktm: Fix php53lint exclusion, and missing php55lint to mw/vendor gate [integration/config] - 10https://gerrit.wikimedia.org/r/267828 [03:44:06] (03CR) 10Legoktm: [C: 032] Fix php53lint exclusion, and missing php55lint to mw/vendor gate [integration/config] - 10https://gerrit.wikimedia.org/r/267828 (owner: 10Legoktm) [03:45:24] (03Merged) 10jenkins-bot: Fix php53lint exclusion, and missing php55lint to mw/vendor gate [integration/config] - 10https://gerrit.wikimedia.org/r/267828 (owner: 10Legoktm) [03:48:03] !log deploying https://gerrit.wikimedia.org/r/267828 [03:48:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [03:58:54] (03PS1) 10Legoktm: Fix php53/php55 regexes. Regex is hard [integration/config] - 10https://gerrit.wikimedia.org/r/267829 [05:33:31] 10Deployment-Systems, 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1988891 (10greg) [06:27:06] Yippee, build fixed! [06:27:06] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #886: 09FIXED in 8 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/886/ [07:32:29] 10Continuous-Integration-Config: translatewiki.net phplint job should allow PHP 5.4 syntax - https://phabricator.wikimedia.org/T97889#1988982 (10Nikerabbit) hhvmlint sounds best since we use it exclusively at translatewiki.net - assuming there are no surprises. [08:47:22] 10Continuous-Integration-Infrastructure: integration-slave-precise-1011.eqiad.wmflabs has not run puppet since Jan 25 - https://phabricator.wikimedia.org/T125474#1989129 (10hashar) 5Open>3Resolved a:3hashar Puppet in /var/log/puppet.log yells about: `Server hostname 'integration-puppetmaster.eqiad.wmflabs'... [08:47:43] !log pooling back integration-slave-precise1011 , puppet run got fixed ( https://phabricator.wikimedia.org/T125474 ) [08:47:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:57:57] I can't reproduce https://integration.wikimedia.org/ci/job/mwext-testextension-php55/8/console locally... only change I can think of is conversion of ULS to extension registration, and maybe change from php53 to php55 [09:05:47] hashar: ^ [09:06:26] FWIW, the dependency of ContentTranslation on UniversalLanguageSelector seems to be defined correctly. [09:06:55] (03CR) 10Hashar: "Follow up is https://gerrit.wikimedia.org/r/#/c/267226/" [integration/config] - 10https://gerrit.wikimedia.org/r/259783 (https://phabricator.wikimedia.org/T120365) (owner: 10Phedenskog) [09:07:51] (03PS2) 10Hashar: Use bash to collect return for different WebPageTest scripts [integration/config] - 10https://gerrit.wikimedia.org/r/267226 (owner: 10Phedenskog) [09:15:04] aharoni: Nikerabbit if you look at the whole console log at https://integration.wikimedia.org/ci/job/mwext-testextension-php55/8/consoleFull [09:15:15] there is a zuul-cloner command that clones multiple repositories [09:15:17] and ULS is not included [09:15:42] hashar: I don't see any extension included... or am I looking wrong lines [09:16:04] and what is to be done to fix it? [09:16:38] maybe dependencies are not injected on the php55 job [09:16:56] we have the dependencies defined in ext_dependencies.py AND in mediawiki-extensions.yaml [09:17:00] can you quickly fill a bug copy pasting the issue + link to your code change please? [09:17:04] and CC legoktm [09:17:06] and I'd say that two places is already too many :) [09:17:07] will look/fix it [09:17:10] filing [09:17:18] but would like something to be kept in Phabricator history [09:17:35] what I think happen is the ext_dependencies.py function is not applied on the php55 jobs [09:17:47] hashar: Continuous-Integration-Config? [09:17:52] the zuul/layout.yaml has some regex to define on which job the parameter function is applied [09:17:54] aharoni: yes [09:17:56] ack [09:17:59] aharoni: cc legoktm as well :} [09:18:05] will look at it in a few minutes [09:19:15] so yeah [09:19:47] aharoni: on the same change, a php53 job passed properly and its parameter list at https://integration.wikimedia.org/ci/job/mwext-testextension-php53/1218/parameters/ shows EXT_DEPENDENCIES has a bunch of content [09:19:56] but there is no such EXT_DEPENDENCIES set for the php55 job [09:20:02] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release8, 7WorkType-Maintenance: ContentTranslation builds fail because of missing UniversalLanguageSelector dependency - https://phabricator.wikimedia.org/T125495#1989198 (10Amire80) 3NEW [09:23:00] yeah [09:23:02] it is broken [09:28:40] 10Beta-Cluster-Infrastructure, 6operations, 7HHVM: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1989210 (10Joe) a:3Joe [09:29:52] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release8, 7WorkType-Maintenance: ContentTranslation builds fail because of missing UniversalLanguageSelector dependency - https://phabricator.wikimedia.org/T125495#1989212 (10hashar) On that change, a previous r... [09:30:09] aharoni: so I have confirmed/found the root cause. No idea how to fix it properly though :-/ [09:30:27] hashar: thanks. Who will know, Lego? [09:32:50] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #702: 15ABORTED in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/702/ [09:33:35] !log restarting Jenkins [09:33:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:33:45] aharoni: I think I will mass revert [09:33:51] dont have time today to do a proper fix :/ [09:34:28] 10Beta-Cluster-Infrastructure, 6operations, 7HHVM: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1989220 (10Joe) And... it's done. tin has been reimaged to trusty as of now. [09:35:05] 10Deployment-Systems, 15User-bd808: Special:Version on Wikimedia wikis shows outdated commit hashes for submodules - https://phabricator.wikimedia.org/T116345#1989223 (10Joe) [09:35:13] 10Beta-Cluster-Infrastructure, 6operations, 7HHVM: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1989221 (10Joe) 5Open>3Resolved [09:38:00] !log Jenkins is fully up and operational [09:38:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:51:16] 10Deployment-Systems, 3Scap3, 6Performance-Team, 6operations, 7HHVM: Make scap able to depool/repool servers via the conftool API - https://phabricator.wikimedia.org/T104352#1989250 (10Joe) [09:51:21] 10Deployment-Systems: [scap] Local sync script on any individual server should be atomic - https://phabricator.wikimedia.org/T22085#1989252 (10Joe) [09:51:23] 6Release-Engineering-Team, 10Wikimedia-Apache-configuration, 6operations: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#1989249 (10Joe) 5Open>3Resolved [10:38:15] Project beta-scap-eqiad build #88348: 04FAILURE in 2 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/88348/ [10:40:08] 10Continuous-Integration-Config, 7WorkType-NewFunctionality: Zuul can only apply one parameter function. Prevent us from injecting both php55 and extension dependencies - https://phabricator.wikimedia.org/T125498#1989298 (10hashar) 3NEW [10:53:14] 10Deployment-Systems, 3Scap3: Build debian package for Scap3 - https://phabricator.wikimedia.org/T125413#1989328 (10fgiunchedi) I've built scap source/binary debian packages and uploaded to trusty/jessie repos (precise is missing on purpose, since we're deprecating it but easy to just copy to precise too) ```... [10:59:39] 10Continuous-Integration-Config: Create generic mwext-jsduck-publish job and use it - https://phabricator.wikimedia.org/T125473#1989345 (10Paladox) [11:01:24] 10Continuous-Integration-Config, 7WorkType-NewFunctionality: Zuul can only apply one parameter function. Prevent us from injecting both php55 and extension dependencies - https://phabricator.wikimedia.org/T125498#1989351 (10Paladox) [11:34:42] 10Deployment-Systems, 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1989404 (10hashar) [11:35:05] Yippee, build fixed! [11:35:05] Project beta-scap-eqiad build #88350: 09FIXED in 7 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/88350/ [12:01:50] hashar_: has the php55 breakage been reverted? I'm about to sleep, and since it didn't actually work for what I needed, we should probably revert if it broke other stuff [12:14:11] 10Deployment-Systems, 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1989465 (10hashar) I got the scap log in a spreadsheet (only WMF folks can read/access though) https://docs.google.com/a/wikimedia.org/... [12:30:56] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989525 (10Joe) 3NEW [12:44:46] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989556 (10Joe) Looking at the audit log on mira, it seems that for some unknown reason ``` /usr/local/bin/scap-master-sync tin ``` was run and that... [12:47:44] 10Deployment-Systems, 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1989558 (10Joe) Please note this needs to be applied to tin too. [12:48:41] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989562 (10hashar) The current situation is: | Host | Dir | Date | Content | tin | /srv/mediawiki-staging/php-1.27.0-wmf.10/ | Feb 2 09:38 | Gone | tin... [12:54:27] Project beta-scap-eqiad build #88358: 04FAILURE in 7 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/88358/ [13:00:31] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989587 (10hashar) Events around 9:38: | 2016-02-02T09:12:10.000Z | mira | INFO | scap.announce | Synchronized wmf-config/db-eqiad.php: Depool db1036,... [13:04:36] Yippee, build fixed! [13:04:36] Project beta-scap-eqiad build #88359: 09FIXED in 8 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/88359/ [13:13:52] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989627 (10Joe) I created a tarball with the last known valid version of the deployed code (ironically, on tin) and on one appserver so that data that w... [13:25:02] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989655 (10hashar) Earliest files on mira:/srv/mediawiki-staging are from 8:39am Lets look at the history on either tin or mira (since they got synced)... [13:28:27] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989656 (10Joe) yes @hashar that's exactly what happened. [13:44:27] 10Continuous-Integration-Infrastructure, 10Math, 10VisualEditor: Math tests failing - https://phabricator.wikimedia.org/T125513#1989675 (10Physikerwelt) 3NEW [13:57:32] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989708 (10Joe) To better clarify the timeline: - I reimaged tin around 8 UTC; - puppet created a git clone of operations/mediawiki-config in /srv/med... [13:58:42] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989711 (10hashar) CC ing the whole #releng . **TL;DR:** we have lost staging area (code / caches / settings) from BOTH deployment servers and have to... [14:06:45] 10Continuous-Integration-Infrastructure, 10Math, 10VisualEditor: Math tests failing - https://phabricator.wikimedia.org/T125513#1989723 (10Esanders) p:5Triage>3Unbreak! [14:20:51] 10Continuous-Integration-Infrastructure, 10Math, 10VisualEditor: Math tests failing - https://phabricator.wikimedia.org/T125513#1989763 (10Physikerwelt) @esanders: do you have access to the jenkins clients? It would be interesting to figure out what has been changed there. For example one could test to compi... [14:28:24] Yippee, build fixed! [14:28:24] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #774: 09FIXED in 2 min 24 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/774/ [15:20:47] 10Continuous-Integration-Infrastructure, 10Math, 10VisualEditor: Math tests failing - https://phabricator.wikimedia.org/T125513#1989919 (10Esanders) I'm not really familiar with it. @Jdforrester-WMF or @hashar will be better placed to fixed this. [15:36:14] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: Backup all of /srv on mira and/or tin (deployment servers) - https://phabricator.wikimedia.org/T125527#1989957 (10jcrespo) 3NEW a:3jcrespo [15:44:49] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: Backup all of /srv on mira and/or tin (deployment servers) - https://phabricator.wikimedia.org/T125527#1989970 (10Dzahn) 26G /srv/ 4.7G /srv/deployment 16K /srv/lost+found 6.5G /srv/mediawiki 15G /srv/mediawiki-staging 1.1M /srv/patches 476K /sr... [15:46:30] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: Backup all of /srv on mira and/or tin (deployment servers) - https://phabricator.wikimedia.org/T125527#1989973 (10demon) Yes, that's the entire problem, mw-staging went missing. At the very least, we want deployment, mw-staging and patches. [15:55:39] 10Deployment-Systems, 6Release-Engineering-Team, 6operations, 5Patch-For-Review: Backup all of /srv on mira and/or tin (deployment servers) - https://phabricator.wikimedia.org/T125527#1990029 (10Dzahn) @demon ok, thanks, i uploaded a patch for the entire /srv @akosiaris would you see 26G as an issue for... [16:15:14] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1990064 (10phuedx) [16:15:31] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1989525 (10phuedx) [16:19:03] 10Deployment-Systems, 6Release-Engineering-Team, 6operations, 5Patch-For-Review: Backup all of /srv on mira and/or tin (deployment servers) - https://phabricator.wikimedia.org/T125527#1990070 (10akosiaris) @Dzahn, no, 26GB is not really an issue but don't use the disk space on helium as a guide. It's the s... [16:19:49] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1990076 (10akosiaris) [16:19:52] 10Deployment-Systems, 6Release-Engineering-Team, 6operations, 5Patch-For-Review: Backup all of /srv on mira and/or tin (deployment servers) - https://phabricator.wikimedia.org/T125527#1990074 (10akosiaris) 5Open>3Resolved Merged, done. Tomorrow we should have the first back up. [16:24:45] 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1990100 (10zeljkofilipin) [16:24:47] 7Browser-Tests, 10Browser-Tests-Infrastructure, 10CirrusSearch, 6Discovery, and 3 others: Upgrade CirrusSearch browser tests to use mediawiki_selenium 1.x - https://phabricator.wikimedia.org/T99653#1990098 (10zeljkofilipin) 5Open>3Resolved [16:25:55] 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1990102 (10zeljkofilipin) 5Open>3Resolved [16:26:00] 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 3 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1990103 (10zeljkofilipin) [16:29:44] 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 6Security, 5MW-1.27-release-notes, and 3 others: Update all repositories that use mediawiki_selenium Ruby gem to version 1.6.x - https://phabricator.wikimedia.org/T114241#1990116 (10zeljkofilipin) 5Open>3Resolved [16:39:49] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1990172 (10Jdforrester-WMF) [16:39:52] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1990171 (10Jdforrester-WMF) [16:44:06] James_F: Heh, that's pretty true yes. [16:44:16] Unbreaking tin/mira is probably a blocker. [16:44:17] :) [16:44:31] ostriches: I jinxed us with https://phabricator.wikimedia.org/T125475#1988640 [16:49:15] James_F: yeah, I cringed when I saw that comment :) [16:49:27] Ah well. [16:50:08] You probably already know, but in case not: I've noticed the qunit test being flaky a few times in the last few days. See https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/28780/consoleFull for example. [16:51:21] anomie: most people's attention is diverted at the moment, can you file a task please? [16:52:12] (03PS1) 10Paladox: Revert "Revert all the php53/php55 recent additions" [integration/config] - 10https://gerrit.wikimedia.org/r/267911 [16:52:51] 10Continuous-Integration-Infrastructure, 10Math, 10VisualEditor: Math tests failing - https://phabricator.wikimedia.org/T125513#1990224 (10Physikerwelt) was this fixed: https://gerrit.wikimedia.org/r/#/c/267611/ [16:54:58] 10Continuous-Integration-Infrastructure: qunit test is being flaky - https://phabricator.wikimedia.org/T125536#1990230 (10Anomie) 3NEW [16:55:08] greg-g: Done [16:55:32] twentyafterfour: Hi could you create this repo https://github.com/wikimedia/operations-software in phabricator please. [16:57:28] anomie: ty [17:06:34] !log running sync-common on mw2051 and mw1119 [17:06:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:19:10] (03PS1) 10Paladox: [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 [17:19:53] (03PS2) 10Paladox: [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) [17:20:04] (03CR) 10jenkins-bot: [V: 04-1] [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) (owner: 10Paladox) [17:20:57] (03CR) 10jenkins-bot: [V: 04-1] [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) (owner: 10Paladox) [17:21:46] (03PS3) 10Paladox: [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) [17:22:39] (03CR) 10jenkins-bot: [V: 04-1] [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) (owner: 10Paladox) [17:24:57] (03PS4) 10Paladox: [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) [17:25:51] (03CR) 10jenkins-bot: [V: 04-1] [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) (owner: 10Paladox) [17:27:09] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1990357 (10hashar) We have restored 1.27.0-wmf.8 1.27.0-wmf.9 and 1.27.0-wmf.10 . Regenerated the l10n cache. mw1017 has been synced. We are now tryin... [17:31:08] 10Deployment-Systems, 10Salt: Provide mechanism to add/remove minions from git-deploy - https://phabricator.wikimedia.org/T74319#1990368 (10ArielGlenn) I've updated my docker salt cluster testebed to work with wmf trebuchet/trigger latest versions (grabs from puppet repo and from gerrit repo respectively): see... [17:31:27] (03PS5) 10Paladox: [operations/software] Add Jenkins tests [integration/config] - 10https://gerrit.wikimedia.org/r/267921 (https://phabricator.wikimedia.org/T125363) [17:34:53] hashar: Could you review https://gerrit.wikimedia.org/r/#/c/267548/ and https://gerrit.wikimedia.org/r/#/c/267541/ and https://gerrit.wikimedia.org/r/#/c/267540/ please. [17:37:01] paladox: unavailable today sorry :-/ [17:37:09] oh ok. [17:42:17] !log cleaning mwext-donationinterfacecore125-testextension-php53 workspace on integration-slave-precise-1013 [17:42:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [17:59:10] 10Continuous-Integration-Config, 6Discovery, 10Kartographer, 10Maps, and 2 others: Add VisualEditor as a CI dependency of the Kartographer extension - https://phabricator.wikimedia.org/T122926#1990451 (10Yurik) [18:00:21] (03PS7) 10Cdentinger: Adds DonationInterface test against mediawiki 1.25 [integration/config] - 10https://gerrit.wikimedia.org/r/266653 (https://phabricator.wikimedia.org/T124105) [18:00:23] (03PS4) 10Cdentinger: add new DI test to experimental pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/267690 (https://phabricator.wikimedia.org/T124105) [18:02:20] 7Browser-Tests, 10MediaWiki-extensions-MultimediaViewer: Fix failed MultimediaViewer browsertests Jenkins jobs - https://phabricator.wikimedia.org/T94157#1990461 (10Jdlrobson) [18:02:29] (03PS5) 10Cdentinger: add new DI test to experimental pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/267690 (https://phabricator.wikimedia.org/T124105) [18:06:05] (03CR) 10Dduvall: [C: 032] add new DI test to experimental pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/267690 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [18:08:49] hashar: Currently here? [18:09:55] Or another CI expert? [18:10:46] Luke081515: hashar is busy with deployment issues atm. what's up? [18:11:30] marxarelli: Check of https://gerrit.wikimedia.org/r/#/c/267780/ did not work. There was a "recheck" input 7 hours ago, but normaly the tests there need less than 10 minutes [18:15:20] Luke081515: hmm, not sure. our whitelist includes MarcoAurelio, so it should have accepted his "recheck" [18:15:24] can you try it yourself? [18:15:47] Ok, I tried it [18:17:46] marxarelli: Strange, works now [18:20:48] (03CR) 10Dduvall: [C: 032] Adds DonationInterface test against mediawiki 1.25 [integration/config] - 10https://gerrit.wikimedia.org/r/266653 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [18:22:56] greg-g: Evening SWAT is at the normal time today? [18:26:10] Luke081515: marxarelli I had to restart Zuul earlier today. It was stuck in some loop apparently [18:26:15] recheck should work now [18:26:41] Luke081515: we will most probably skip swat and mw new version deployment entirely [18:27:18] hm, ok [18:31:46] (03Merged) 10jenkins-bot: Adds DonationInterface test against mediawiki 1.25 [integration/config] - 10https://gerrit.wikimedia.org/r/266653 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [18:31:48] (03Merged) 10jenkins-bot: add new DI test to experimental pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/267690 (https://phabricator.wikimedia.org/T124105) (owner: 10Cdentinger) [18:32:53] !log Reloading Zuul to deploy If1f3cb60f4ccb2c1bca112900dbada03a8588370 [18:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [18:33:08] cwd: ^ [18:33:41] marxarelli: thanks a million! [18:33:43] you can trigger your new job via "check experimental" in gerrit [18:33:45] np! [18:42:42] \O/ [19:31:01] thcipriani: So, I'm wondering if rsync is the best way to sync that data between masters. [19:33:57] 10Continuous-Integration-Infrastructure, 10Math, 10VisualEditor: Math tests failing - https://phabricator.wikimedia.org/T125513#1991003 (10Jdforrester-WMF) I'm merging this with {T125495} because I think they're all the same issue. I think. [19:34:03] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release8, 5Patch-For-Review, 7WorkType-Maintenance: ContentTranslation builds fail because of missing UniversalLanguageSelector dependency - https://phabricator.wikimedia.org/T125495#1989198 (10Jdforrester-WMF) [19:34:08] 10Continuous-Integration-Infrastructure, 10Math, 10VisualEditor: Math tests failing - https://phabricator.wikimedia.org/T125513#1991005 (10Jdforrester-WMF) [19:36:37] ostriches: would be nice to have a flattened got repo, actually. [19:37:58] You mean version php-* and subrepos all as one in the parent? [19:40:19] Yeah. Had this thought a few times. Makes a lot of stuff possible. [19:49:40] so [19:50:00] thcipriani: ostriches: marxarelli: greg-g: with all the staging mess today, I havent cut the wmf.12 branche [19:50:18] which mean there is zero chance the mediawiki train is going to start 10 minutes from now :/ [19:50:57] plus side, today was an excellent catch up exercise. I am more familiar with our setup / scap usage [19:51:03] * ostriches starts branching [19:51:17] guess that means we do the train [19:51:27] anomie had a few patch he needed in [19:51:45] and I haven't checked whether the session manager got reverted in master [19:51:46] ostriches: i can take it. i'm technically on deck to deploy anyway [19:52:00] sorry late notice /screw up [19:52:05] been a busy day [19:52:28] the session manager revert would be https://gerrit.wikimedia.org/r/#/c/267737/ [19:52:35] we probably want to sync with #mediawiki-core folks [19:53:08] Can you ping me, if train arrived at group0? [19:53:19] marxarelli: What's another tab? :) [19:53:34] asked them to join in [19:54:26] ostriches: we got integration-make-end-branch in a good working state, FYI. [19:54:30] hashar: that's the re-revert [19:54:41] the revert was already merged into master afaik [19:54:44] ohh f$$$$ [19:54:45] marxarelli: I have it running now. [19:55:06] It's doing all the extensions now [19:55:08] ostriches: kk [19:55:29] ostriches: you doing it locally? i don't see you on integration-make-wmf-branch [19:55:43] ostriches: they landed patches a few minutes ago that are required for the new branch [19:55:54] Oh, I always do it locally lol. [19:55:58] namely https://gerrit.wikimedia.org/r/#/c/267734/ and https://gerrit.wikimedia.org/r/#/c/267735/ [19:56:01] bad habits die hard :p [19:56:53] ostriches: also can't we cut branch using the Gerrit API ? [19:57:10] Why write new tools for gerrit? [19:57:13] greg-g: is it possible to merge beta cluster only config changes today, or is that also disallowed today [19:57:14] hashar: hi [19:57:21] you can restart on integration-make-wmf-branch using the --continue-from branch. [19:57:38] thcipriani: Well, only if my local clone fails :p [19:57:46] anomie: ostriches started cutting the branch a few minutes. Will have to double check your $wgAuthenticationTokenVersion patches landed [19:57:55] then hopefully we will get to deploy it ;) [19:57:56] hashar: Gerrit says they're merged [19:58:14] yeah [20:03:59] ostriches: this config patch that was scheduled for swat needs to go with/before .12 -- https://gerrit.wikimedia.org/r/#/c/267807/ [20:04:16] k [20:04:20] without it the hhvm logs will be full of spam about undefined vars [20:04:48] * bd808 returns to his regularly scheduled lunch [20:11:15] Failed for me on RandomRootPage [20:11:21] marxarelli: If you wanted to resume :p [20:11:30] ! [remote rejected] wmf/1.27.0-wmf12 -> wmf/1.27.0-wmf12 (prohibited by Gerrit) [20:11:58] ostriches: oh bummer. yeah, i can take it [20:12:18] acl looks right... [20:14:14] thcipriani: so `./make-wmf-branch -n 1.27.0-wmf.12 -o master --continue-from RandomRootPage` right? [20:15:46] marxarelli: yup should be correct. [20:16:09] wee [20:16:13] ostriches: i get the same error [20:17:02] master? [20:17:06] I thought -o was the old branch [20:17:30] it is. We cut from master every time though. [20:18:27] marxarelli: did you remember to key forward? [20:18:35] thcipriani: yep [20:18:39] hmm [20:18:52] * ostriches suggests https [20:18:58] instead of key forwarding [20:21:27] lemme try this... [20:21:52] gonna see if I get the same error marxarelli ostriches [20:22:06] k [20:22:46] s/lemme try this/here hold my beer/ [20:22:49] fixed for you [20:23:44] :) [20:24:19] ostriches: wait, so you did -o 1.27.0-wmf.11? [20:24:34] No wmf.10 [20:24:43] -o wmf.10 -n wmf.12 [20:24:49] I tried to be clever? [20:25:17] that's not good: that would mean that it just cloned wmf.10 and declared it a new branch [20:25:25] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1991389 (10hashar) Servers are being synced in small batches. We are proceeding with canary servers first and so far it is going fine. [20:25:46] also: same error: error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/RandomRootPage.git' [20:27:09] so couple things: 1 how can we cleanup the extensions that have already been branched so that it cuts a new branch from master and 2. why can't we push to RandomRootPage via ssh? [20:27:29] thcipriani, ostriches: i'm looking at MobileFrontend and it's branched from master [20:27:43] ah [20:27:48] RandomRootPage has been removed [20:27:50] archived [20:27:58] it has been merged in mediawiki-core [20:28:00] Ah, it was still in the branching config. [20:28:07] rm, commit, move on :) [20:28:10] and the mediawiki-release script got updated (pretty sure ) [20:29:20] ah oldVersion is just for master [20:29:38] ostriches: k. removed. i'll continue on integration-make-wmf-branch [20:29:47] for some horrible reason... [20:32:10] bd808: You said before is fine? [20:33:09] ostriches: yeah. we were going to do it last night but the scap in swat ran us out of time [20:33:50] k [20:34:16] ostriches, thcipriani: branching again ... [20:35:53] kk [20:36:42] (03PS1) 10Dduvall: Remove RandomRootPage extension from branching config [tools/release] - 10https://gerrit.wikimedia.org/r/267989 [20:41:44] crap. i'm getting a bunch of "unable to checkout submodule" [20:42:36] oh, probably because i used continue-from on the first pass [20:42:38] thcipriani: ^ ? [20:43:23] marxarelli: paste? [20:44:45] thcipriani: https://tools.wmflabs.org/paste/view/f6f3208e [20:45:05] * thcipriani looks [20:45:07] the submodules weren't cloned [20:45:22] because i used continue-from on the initial run [20:45:41] shouldn't matter, should be cloning from upstream [20:46:23] there's nothing in /var/make-wmf-branch/wmf for AbuseFilter [20:46:38] https://github.com/wikimedia/mediawiki-extensions-AccountAudit [20:46:49] ^ doesn't seem like that extension was branched :\ [20:47:15] yeah, but it clones twice, one of the things we wanted to fix but never have [20:47:32] so it's still trying to add the submodule from gerrit, but it's trying to clone a branch that doesn't exist... [20:48:00] ah, i see [20:48:12] yeah, you're right. there are a bunch of these errors [20:48:14] wtf [20:48:32] ostriches: were there failures from your make-wmf-branch run? [20:49:04] thcipriani: updated the paste [20:49:48] marxarelli: don't know if I see the difference? [20:49:58] thcipriani: https://tools.wmflabs.org/paste/view/6d4cc5da [20:50:03] nm. it's a new one [20:50:25] and i killed the process, so there's that [20:50:26] Just the original random root page one [20:51:49] ostriches: does your make-wmf-branch/config.json contain AccountAudit? [20:51:50] looks like the branch is missing a . [20:51:53] https://github.com/wikimedia/mediawiki-extensions-EventBus/tree/wmf/1.27.0-wmf12 [20:52:06] oh noes! [20:52:09] instead of wmf/1.27.0-wmf.12 [20:52:47] oops [20:52:53] :-\ [20:53:06] well, i can do the branch cut again and bail after it gets to the one i started from [20:53:14] marxarelli: It does, yes [20:53:30] ostriches: ^ looks like a missing [20:53:33] '.' [20:53:38] ironic [20:54:10] i guess we need a --stop-after arg :) [20:55:45] heh, yeah, I'd say we need to modify the script. So, remove all the extensions after RandomRootPage, and remove branchWmf from the makeWmfBranch.php script [20:55:56] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release8, 5Patch-For-Review, 7WorkType-Maintenance: ContentTranslation builds fail because of missing UniversalLanguageSelector dependency - https://phabricator.wikimedia.org/T125495#1991534 (10hashar) 5Open... [20:56:06] thcipriani: too late [20:56:11] :) [21:10:03] ostriches, thcipriani: ok, branch cut is truckin again. just finishing up core [21:10:42] marxarelli: awesome! [21:15:24] \O/ [21:35:55] legoktm: hey are you back around ? [21:37:26] legoktm: guess you will catch the message anyway. Sorry to have reverted the php5.5 stuff :-/ The dependencies were no more injected because Zuul only accept a single parameter function. Filled details at https://phabricator.wikimedia.org/T125498 [21:37:27] :( [21:37:40] such a pity, cause I was really happy to see php5.5 jobs being introduced [22:04:11] hashar: no worries :) I'll take a look at that today [22:08:34] (03CR) 10Thcipriani: [C: 032] Remove RandomRootPage extension from branching config [tools/release] - 10https://gerrit.wikimedia.org/r/267989 (owner: 10Dduvall) [22:09:18] (03Merged) 10jenkins-bot: Remove RandomRootPage extension from branching config [tools/release] - 10https://gerrit.wikimedia.org/r/267989 (owner: 10Dduvall) [22:24:21] 10Deployment-Systems, 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1991900 (10hashar) The scap log was really {T125506} With tin/mira reimaged, the shebang php5 means it is using PHP 5.5.9 from Trusty... [22:28:26] hashar: am I too late? I’m hoping someone can explain why this is failing: hashar: am I too late? I’m hoping someone [22:28:30] whoah c/p fail [22:28:36] This: https://gerrit.wikimedia.org/r/#/c/268012/ [22:28:49] andrewbogott: for me ? yeah too late sorry [22:28:53] been around for too long [22:28:54] ok [22:28:59] no problem, go sleep :) [22:29:16] also we froze deployment :/ [22:29:30] 10Continuous-Integration-Config, 10MediaWiki-extensions-ContentTranslation, 5ContentTranslation-Release8, 5Patch-For-Review, 7WorkType-Maintenance: ContentTranslation builds fail because of missing UniversalLanguageSelector dependency - https://phabricator.wikimedia.org/T125495#1991922 (10Legoktm) Sorry... [22:29:47] andrewbogott: added myself as a reviewer, but I am unlikely to look at it anytime soon :/ [22:29:52] ok [22:30:10] probably I just don’t know how to read the test output [22:30:25] andrewbogott: bad timing, it'll be fixed in ~5 minutes if you comment "recheck" [22:30:33] oh, ok! [22:30:34] thanks [22:34:36] 10Deployment-Systems, 6operations, 7HHVM: HHVM lock-ups - https://phabricator.wikimedia.org/T89912#1991956 (10ori) 5Resolved>3Open Still happens occasionally. [22:48:30] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1992022 (10greg) a:3hashar Assigning to hashar since he's running the train (tomorrow). [22:49:19] 6Release-Engineering-Team: MW 1.27.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T125596#1992030 (10greg) 3NEW [22:49:33] 6Release-Engineering-Team: MW 1.27.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T125597#1992037 (10greg) 3NEW [22:53:36] 6Release-Engineering-Team: MW 1.27.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T125596#1992053 (10greg) [22:54:42] 6Release-Engineering-Team: MW 1.27.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T125596#1992058 (10greg) a:3hashar Assigning to @hashar as he'll be running this train. [23:01:29] 7Browser-Tests, 10MediaWiki-Authentication-and-authorization: Create some end-to-end tests for SessionManager - https://phabricator.wikimedia.org/T125599#1992090 (10greg) 3NEW a:3Anomie [23:06:56] 6Release-Engineering-Team, 5WMF-deploy-2016-02-09_(1.27.0-wmf.13): MW 1.27.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T125596#1992138 (10Luke081515) [23:11:22] Yippee, build fixed! [23:11:22] Project browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #405: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/browsertests-Gather-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/405/ [23:11:22] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 3Discovery-Portal-Sprint: Automatically deploy Wikimedia portals to Beta Cluster - https://phabricator.wikimedia.org/T124848#1992161 (10debt) p:5Triage>3Normal [23:13:25] 10Deployment-Systems, 6Release-Engineering-Team, 6operations: /srv/mediawiki-staging broken on both scap masters - https://phabricator.wikimedia.org/T125506#1992199 (10demon) So, all apaches are back in sync with master now. Couple of actionables here off the top of my head: # Backups (done in child T125527)... [23:15:00] 6Release-Engineering-Team, 5WMF-deploy-2016-02-16_(1.27.0-wmf.14): MW 1.27.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T125597#1992212 (10Luke081515) [23:20:36] 10Deployment-Systems, 6Release-Engineering-Team: refreshCdbJsonFiles in scap fails on mira due to missing dba_open function in hhvm - https://phabricator.wikimedia.org/T125477#1992231 (10bd808) In addition to the ICU mismatch, the PHP 5.5 version we get by default on trusty is subject to a glibc iconv bug: {T3... [23:23:53] (03PS1) 10Cdentinger: Make new DI job live, set old one to non-voting [integration/config] - 10https://gerrit.wikimedia.org/r/268021 (https://phabricator.wikimedia.org/T124105) [23:34:36] oh, thcipriani I wanted to say explicitly: good work on https://phabricator.wikimedia.org/T125468 yesterday [23:35:02] greg-g: I revert with the best of 'em [23:35:04] 6Release-Engineering-Team: MW 1.27.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T125475#1992265 (10greg) >>! In T125475#1988640, @Jdforrester-WMF wrote: > Tentatively closable, then? never say this again :P [23:53:48] 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team, 6Discovery, 3Discovery-Portal-Sprint: Automatically deploy Wikimedia portals to Beta Cluster - https://phabricator.wikimedia.org/T124848#1992353 (10debt)