[00:00:54] My answer to "why not?" about +2 in mw-config without deploy is "because Roan will yell at you!" ;) [00:01:07] haha [00:01:23] I feel like Chad took over yelling at people about that years ago [00:05:37] and now icinga does it for us, yelling as a service [00:09:50] jdlrobson: I am now finally cherry-picking your hamburger menu patch, it took forever to merge because of the unit test flakiness issues [00:10:16] I also merged your patch for those issues so it would let me merge this [00:20:06] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:22:56] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:48:42] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3188555 (10TerraCodes) [01:31:08] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:32:44] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3188702 (10TerraCodes) [02:09:49] RoanKattouw: thanks for the explanation, makes sense, I didn't know about "must immediately deploy" rule [02:14:50] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3188740 (10TerraCodes) [02:35:45] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3188759 (10TerraCodes) [02:56:32] 10Scap (Scap3-Adoption-Phase1), 10scap2, 07WorkType-NewFunctionality: Need a way to see config diffs in Scap - https://phabricator.wikimedia.org/T118206#3188768 (10mmodell) [05:51:51] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3188889 (10TerraCodes) [05:52:09] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#2420118 (10TerraCodes) [07:39:37] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3189095 (10TerraCodes) a:03TerraCodes [08:31:31] 05Gitblit-Deprecate, 13Patch-For-Review: Fix references to git.wikimedia.org in all repos - https://phabricator.wikimedia.org/T139089#3189218 (10TerraCodes) [10:07:21] !log upgrade swift to 2.2.0 on deployment-ms* [10:07:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:49:08] (03PS1) 10Hashar: mediawiki/skins/Poncho: add tests [integration/config] - 10https://gerrit.wikimedia.org/r/348706 [11:51:22] (03CR) 10Hashar: [C: 032] mediawiki/skins/Poncho: add tests [integration/config] - 10https://gerrit.wikimedia.org/r/348706 (owner: 10Hashar) [11:52:36] (03Merged) 10jenkins-bot: mediawiki/skins/Poncho: add tests [integration/config] - 10https://gerrit.wikimedia.org/r/348706 (owner: 10Hashar) [12:03:06] (03PS2) 10Hashar: [CodeMirror] Express dependency on VisualEditor and WikiEditor [integration/config] - 10https://gerrit.wikimedia.org/r/348640 (owner: 10Jforrester) [12:03:31] (03CR) 10Hashar: [C: 032] [CodeMirror] Express dependency on VisualEditor and WikiEditor [integration/config] - 10https://gerrit.wikimedia.org/r/348640 (owner: 10Jforrester) [12:04:47] (03Merged) 10jenkins-bot: [CodeMirror] Express dependency on VisualEditor and WikiEditor [integration/config] - 10https://gerrit.wikimedia.org/r/348640 (owner: 10Jforrester) [12:20:13] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy: mediawiki_selenium should document SauceLabs usage - https://phabricator.wikimedia.org/T98331#3189549 (10Rammanojpotla) Is this issue to add the documentation of the sauce labs above present in the link above to the documentation of mediawiki-selenium(... [12:50:17] Hey, Can someone make me an owner in https://github.com/wikimedia/data-values-value-view ? I want to release in github so packagist pick it up. hashar ? [12:51:01] (you are owner in wikimedia org) [13:16:59] !log upgrade deployment-jobrunner02 to hhvm 3.18.2+wmf2 - T162354 [13:17:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:17:07] T162354: Frequent TCP RST on connections between HHVM and Redis - https://phabricator.wikimedia.org/T162354 [13:18:08] Project beta-scap-eqiad build #151463: 04FAILURE in 3 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/151463/ [13:25:37] (03PS1) 10Hashar: Semantic: switch to unit tests (non voting) [integration/config] - 10https://gerrit.wikimedia.org/r/348720 [13:27:24] Yippee, build fixed! [13:27:24] Project beta-scap-eqiad build #151464: 09FIXED in 2 min 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/151464/ [13:38:32] (03CR) 10Hashar: "Reviewed the zuul layout diff, it only adds mwext-testextension-hhvm-jessie-non-voting to the test pipeline. I am converting the other re" [integration/config] - 10https://gerrit.wikimedia.org/r/348720 (owner: 10Hashar) [13:46:38] (03PS2) 10Hashar: Semantic: add non voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/348720 [13:49:50] (03PS6) 10Hashar: test: mw ext/skins have test templates in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/332890 [13:51:22] (03CR) 10Hashar: [C: 032] Semantic: add non voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/348720 (owner: 10Hashar) [13:51:46] (03CR) 10Hashar: [C: 032] "All skins / extensions have tests finally. https://gerrit.wikimedia.org/r/#/c/348720/ is the last batch for the Semantic* extensions." [integration/config] - 10https://gerrit.wikimedia.org/r/332890 (owner: 10Hashar) [13:52:46] (03Merged) 10jenkins-bot: Semantic: add non voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/348720 (owner: 10Hashar) [13:53:09] (03Merged) 10jenkins-bot: test: mw ext/skins have test templates in Zuul [integration/config] - 10https://gerrit.wikimedia.org/r/332890 (owner: 10Hashar) [14:08:17] 10Continuous-Integration-Config, 10Analytics, 10Analytics-Dashiki, 13Patch-For-Review: Add CI job for Dashiki - https://phabricator.wikimedia.org/T148019#3189797 (10hashar) 05Open>03declined [14:09:55] !log integration: upgrade puppet on Jessie permanent slaves 3.7.2 -> 3.8.5 (and add ruby-rgen). Done via: salt -v '*' pkg.upgrade [14:09:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:12:02] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:12:54] PROBLEM - Puppet run on saucelabs-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:14:40] PROBLEM - Puppet run on deployment-memc05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:14:48] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:15:49] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:16:28] moauahah [14:16:35] so I have dropped puppet from the puppet master :/ [14:17:29] thats one way to do things [14:17:39] PROBLEM - Puppet run on saucelabs-02 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0] [14:18:21] PROBLEM - Puppet run on jenkinstest is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:18:47] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:22:00] PROBLEM - Puppet run on integration-publishing is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:23:24] PROBLEM - Puppet run on integration-slave-jessie-android is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:24:02] PROBLEM - Puppet run on castor is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:24:24] PROBLEM - Puppet run on integration-slave-docker-1000 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:24:24] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:24:40] RECOVERY - Puppet run on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:05] !log unbreaking integration puppetmaster. Broke it when upgrading the puppet package :( [14:29:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:30:21] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:30:31] PROBLEM - Puppet run on integration-puppetmaster01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:40:30] RECOVERY - Puppet run on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:41:08] hashar hi, im wondering what failure did you get with upgrading puppet? It may have been similar to an error that i got but has since been fixed. [14:41:17] 10Continuous-Integration-Config, 10Analytics, 10Analytics-Dashiki, 13Patch-For-Review: Add CI job for Dashiki - https://phabricator.wikimedia.org/T148019#3189884 (10mforns) Thanks @hashar anyway! [14:41:25] paladox: apt-get dist-upgrade uninstalled puppet entirely [14:41:31] Oh [14:42:04] apt-get update && apt-get upgrade should update puppet. [14:42:52] RECOVERY - Puppet run on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [14:43:22] RECOVERY - Puppet run on jenkinstest is OK: OK: Less than 1.00% above the threshold [0.0] [14:44:24] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:20] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [14:45:50] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [14:47:05] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [14:47:39] RECOVERY - Puppet run on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:48:23] RECOVERY - Puppet run on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0] [14:48:45] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:49:01] RECOVERY - Puppet run on castor is OK: OK: Less than 1.00% above the threshold [0.0] [14:49:27] RECOVERY - Puppet run on integration-slave-docker-1000 is OK: OK: Less than 1.00% above the threshold [0.0] [14:49:51] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [14:51:00] hashar also remeber that java 8 i was talking about with ssh slaves. Turns out that it actually downloads and install that java version if none are detected. [14:57:02] RECOVERY - Puppet run on integration-publishing is OK: OK: Less than 1.00% above the threshold [0.0] [15:03:50] 10Continuous-Integration-Config, 06Reading-Web-Backlog, 10VisualEditor: Intermittent failures of mwext-qunit-jessie - https://phabricator.wikimedia.org/T163123#3186782 (10Jdforrester-WMF) In T163211 I've requested a way to make unit test failures of this sort more apparent, though I'm not sure it's possible. [15:08:01] PROBLEM - Puppet run on integration-publishing is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:22:46] thcipriani|afk: hashar: I'll have to miss today's meeting, in the tech quarterly reviews [15:24:38] mobrovac: np, thanks for the heads-up [15:28:02] RECOVERY - Puppet run on integration-publishing is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:00] PROBLEM - Puppet run on integration-publishing is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:45:44] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy: audit/update headers in files - https://phabricator.wikimedia.org/T69141#3190113 (10Rammanojpotla) @zeljkofilipin should the license headers present in all the files containing the string 'qa-browsertests' are to be removed (as in the link given by you... [15:59:03] RECOVERY - Puppet run on integration-publishing is OK: OK: Less than 1.00% above the threshold [0.0] [17:12:51] (03PS1) 10Gergő Tisza: Add standard tests for testing-access-wrapper library [integration/config] - 10https://gerrit.wikimedia.org/r/348770 [17:38:52] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy: mediawiki_selenium should document SauceLabs usage - https://phabricator.wikimedia.org/T98331#3190547 (10zeljkofilipin) @Rammanojpotla correct. [[ https://phabricator.wikimedia.org/diffusion/MSEL/browse/master/README.md | README.md ]] should have a sen... [17:45:22] 10Beta-Cluster-Infrastructure, 10Flow, 06Collaboration-Team-Triage (Collab-Team-Q3-Jan-Mar-2017), 05MW-1.27-release (WMF-deploy-2015-11-17_(1.27.0-wmf.7)), and 2 others: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#3190580 (10C... [17:53:51] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy: audit/update headers in files - https://phabricator.wikimedia.org/T69141#3190602 (10zeljkofilipin) @Rammanojpotla (I think) license headers should be: - **removed** - if the rest of the files in the repository **do not** have license headers - **updat... [18:17:24] RainbowSprinkles hi, it seems that the gerrit nagios checks will succeed even if gerrit hasent started. Should that be improved so that it will only pass if gerrit actually started and you can access it from the webui? [18:18:24] It checks if the process is running. And we check for apache running. That's sufficient. [18:18:48] ok [19:43:52] (03PS4) 10Hashar: castor: cache binaries on a per job basis [integration/config] - 10https://gerrit.wikimedia.org/r/346152 (https://phabricator.wikimedia.org/T159591) [19:46:01] (03CR) 10Hashar: "I have split the cache in two:" [integration/config] - 10https://gerrit.wikimedia.org/r/346152 (https://phabricator.wikimedia.org/T159591) (owner: 10Hashar) [19:46:08] 10Continuous-Integration-Infrastructure, 10WikimediaUI Style Guide: stylelint-config-wikimedia: Add rule for two or more newline before eof - https://phabricator.wikimedia.org/T163136#3191276 (10Jdforrester-WMF) 05Open>03Invalid We're currently tracking issues on https://www.mediawiki.org/wiki/Manual_talk:... [20:33:13] RainbowSprinkles: https://phabricator.wikimedia.org/T148958#3173999 This and the linked task from that comment are both fixed now, may be worth a minor release? [20:33:48] REL1_27 using Memcached has fatals from Interwiki.php, and REL1_27 using WinCache also has fatals for any cache access. [20:34:07] s/REL1_17/latest 1.27 release/, it's fixed in the branch [20:34:51] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests, 05MW-1.27-release: REL1_27 qunit karma:main tests pass intermittently - https://phabricator.wikimedia.org/T162419#3191510 (10Krinkle) 05Open>03Resolved a:03Krinkle > See https://gerrit.wikimedia.org/r/#/c/329775/ Merged. [20:36:49] legoktm: What is the status of https://phabricator.wikimedia.org/T138370#2745479 - according to the latest comment it's not a 1.27 issue, but a 1.26 issue, which mismatches the original task description. [20:40:15] hashar hi, i've found a ppa for openjdk 8 we could use on trusty [20:40:27] its openjdk not the oracle one [20:40:39] https://launchpad.net/~openjdk-r/+archive/ubuntu/ppa [20:46:04] paladox: we are going to phase out trusty anyway :] [20:46:06] dont bother! [20:46:14] Ok [20:47:28] * hashar vanishes [20:48:30] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [20:50:10] (03CR) 10Krinkle: [C: 04-1] castor: cache binaries on a per job basis (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/346152 (https://phabricator.wikimedia.org/T159591) (owner: 10Hashar) [21:17:09] Krinkle: Blah. Who needs caching? [21:17:29] Or 1.27 :p [21:46:56] 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations, 10Traffic, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191871 (10Krinkle) 05Open>03Resolved a:03Krinkle Landed in master for 1.28 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations, 10Traffic, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191909 (10demon) There's probably some extensions that need fixing here too. The Elastica library looks possibly... [21:55:24] 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations, 10Traffic, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191910 (10demon) 05Resolved>03Open [22:05:20] 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations, 10Traffic, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191940 (10demon) 05Open>03Resolved >>! In T140658#3191909, @demon wrote: > There's probably some extensions t... [22:05:34] 06Release-Engineering-Team, 10MediaWiki-General-or-Unknown, 06Operations, 10Traffic, and 5 others: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#3191943 (10demon) [22:05:57] Krinkle: Ok, I was thinking we still had extensions being affected by ^, but looks like we're good [22:06:03] Puppet & friends long since fixed too [22:06:59] https://phabricator.wikimedia.org/tag/mw-1.23-release/ - 1 task (String warning) [22:07:18] https://phabricator.wikimedia.org/tag/mw-1.26-release/ - 1 task https://phabricator.wikimedia.org/T100085 still an issue in 1.27/1.28 [22:07:37] 1.26 is past EOL [22:07:48] Yeah, was just going through [22:07:55] but that one is still in 1.27 and master as well [22:07:58] regression since 1.26 [22:08:05] didn't seem right to untag [22:08:10] but the tag is archived now [22:08:43] We have tags for the supported versions, went ahead and remove 1.26 tag [22:11:21] k [22:12:35] I guess we can backport. Just confusing as to whether we release since there's no consistent policy on backports of bugs [22:12:57] Which we should probably fix at some point. [22:16:06] RainbowSprinkles: Would you happen to know why tin and terbium run an older version of PHP/HHVM than the appservers do? [22:16:31] I would not [22:16:36] This made for a confusing afternoon because things that fail in prod suddenly didn't fail when I tried them in eval.php, until I went to mwdebug1002 and used eval.php there :/ [22:16:45] OK, then I'll file a task [22:16:59] They should already be on jessie, so that's not it [22:17:12] But yeah, good question! [22:17:53] I'd CC Filippo [22:18:09] OK [22:18:34] Hmm, what's the preferred trick to run php --version on all hosts and gather the results? salt is root-only, but I think mutante was using a different tool for this the other day [22:20:32] I think I'm going to need this, because some app servers are also running the old vresion [22:20:49] mw1220 is old, but mwdebug1001/1002 are new [22:21:29] RainbowSprinkles: This explains why the Echo unserialize errors you were complaining about a month ago are so weird BTW, they appear to depend on which appserver you hit as well [22:21:50] I guess salt and/or cumin [22:21:59] Or being super patient with SSHing 100s of places :p [22:22:10] cumin, that was it [22:22:39] cumin is the salt replacement [22:22:44] Basically a wrapper around clusterssh [22:24:14] Meh cumin also requires root [22:24:23] And dsh doesn't work any more because it requires agent forwarding, which is disabled [22:24:43] I suppose I can do a DIY dsh locally [22:25:19] RoanKattouw: https://github.com/Krinkle/dotfiles/blob/master/hosts/KrinkleMac/bin/dsh-ci-slaves [22:25:31] Havent' used since 2015 but may be reclycable to some degree [22:29:05] Hmph it seems I can't ssh into most of these boxes at all [22:30:31] RoanKattouw: You could use DSH with the deployment key with keyholder [22:31:19] How do I do that? [22:31:23] SSH_AUTH_SOCK=/run/keyholder/proxy.sock dsh -F 20 -M -g mediawiki-installation -r ssh -o -oUser=mwdeploy -- foo [22:31:43] Thanks [22:31:48] You can ssh everywhere as mwdeploy using SSH_AUTH_SOCK :) [22:33:12] Yay that's working, thanks [22:33:16] yw [22:33:34] deep down, that's how scap + dsh + keyholder come together to execute your commands too [22:35:36] I wonder if a `scap exec` command could be useful. Let you execute an arbitrary remote command on your targets as the deploy user [22:35:46] Basically wraps that magic ^^ [22:37:56] Not like it's a privilege escalation, it's already possible to do (and easier than writing a plugin for just a one-off thing you wanna do) [22:40:20] I mean it'd run as the mwdeploy user anyway [22:44:05] Yeah, I'm just thinking if it'd be generally useful [22:48:12] RainbowSprinkles: What is naos.codfw.wmnet? [22:48:18] Replacement for mira [22:48:22] (which is dying) [22:48:34] Oh, new deployment server [22:48:46] Yeah, new codfw deployment server. Mira's suffering a hardware failure [22:49:56] (03PS1) 10Chad: Remove delete-stale-branch: use scap clean instead [tools/release] - 10https://gerrit.wikimedia.org/r/348870 [22:57:32] RainbowSprinkles: OK, I've filed T163278 and CCed Filippo as you recommended. Any tags I should tag this with aside from "Operations"? [22:57:33] T163278: Four different PHP/HHVM versions on the cluster - https://phabricator.wikimedia.org/T163278 [22:57:51] RoanKattouw: HHVM, Ops [22:59:40] Ah, dur [22:59:43] Prolly related to T158176 [22:59:43] T158176: Build / migrate to HHVM 3.18 - https://phabricator.wikimedia.org/T158176 [22:59:53] (partially migrated, testing?) [23:03:02] That doesn't explain the 3.12.x differences, but does at least explain 3.18.x vs 3.12.x [23:04:30] Nor the 3.18.x differences for that matter [23:04:56] Aha, "There were some stat_cache related bugs, which are fixed now. 3.18.2 is running on the mediawiki canaries, but the wider rollout is held back until after the DC switchover." [23:05:45] updating mwdebug to 3.18.2 from .1 would be nice [23:22:28] OK, WTF, this just got weirder [23:22:39] It looks like the php --version discrepancies do not explain what I'm seeing [23:23:17] I have a snippet of code that when I run it on terbium through mwscript eval.php mediawikiwiki it works, but if I run it on terbium through sudo -u www-data php /srv/mediawiki/multiversion/MWScript.php eval.php mediawikiwiki it throws an exception [23:23:22] I thought those two were the same? [23:24:12] Ooooh, mwscript uses php5 [23:24:19] Which is Zend, not HHVM [23:24:57] Yeha [23:25:03] It's always done that [23:25:20] T146285 [23:25:21] T146285: Switch mwscript from Zend PHP5 to default php alternative (egHHVM) - https://phabricator.wikimedia.org/T146285 [23:25:22] So it turns out that I've found data that unserializes cleanly in Zend but not HHVM [23:25:38] Which is the core the Echo issue [23:25:49] The great thing is that that means that I can write a maintenance script that reserializes them [23:25:59] And that'll work because mwscript uses Zend to execute maintenance scripts [23:26:20] But that's terrible, because maintenance scripts should not be run by a different engine [23:26:27] Oh, I see, the task you referenced is about that [23:26:28] Yep, you're totally right [23:26:33] It's been that way forever. We need to fix it [23:28:32] So, the "scap was failing on PHP calls" issue that caused the hardcoding is probably gone (we don't have any PHP scripts we call anymore) [23:28:45] Don't fix it yet ;) [23:28:46] The only thing we do in PHP in scap anymore is shelling out to php -l [23:28:53] I need to exploit it for my reserialization script [23:29:10] Which, iirc is impossibly shitty in hhvm's `php` cli. [23:29:37] I have DB rows with PHP-serialized data that don't unserialize in HHVM but do in Zend, and because a __sleep() method was introduced later, just reserializing them is enough to fix that issue [23:29:43] The startup would probably be unacceptably slow [23:29:52] If you're doing sync-dir on a folder with a crapton of PHP in it [23:30:04] So I need to write a maintenance script that queries those rows from the DB, unserializes them, serializes them again, and writes the result back [23:30:13] And that script will need to run in Zend, not HHVM [23:30:36] I mean I'm not fixing it today :p [23:30:40] I'm just saying, should be fixed [23:30:47] The inconsistency is annoying [23:30:53] Yeah it should be [23:30:56] (but the overhead of hhvm just for a quick script is also kinda lame) [23:31:17] I'm also curious about overhead for hhvm -l vs php -l in scap [23:31:21] On a sufficiently large directory [23:32:38] Actually, we already lint with hhvm [23:32:41] We use php, not php5 [23:32:42] :p [23:42:15] RoanKattouw: Just read your treatise on T163220. What a rabbit hole for a stupidly simple bug/fix. [23:42:16] T163220: Add Icinga check for CPU frequency on Dell R320 - https://phabricator.wikimedia.org/T163220 [23:42:23] Blah, meant T159372 [23:42:24] T159372: Flow: DateTimeZone::__construct(): Unknown or bad timezone (+00:00) - https://phabricator.wikimedia.org/T159372 [23:46:28] Yup :) [23:46:32] * :/