[00:23:22] 10Beta-Cluster-Infrastructure: Account creation throttling too restrictive on Beta Cluster - https://phabricator.wikimedia.org/T87704#2991851 (10Tgr) [00:23:24] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Don't throttle WMF office IP(s) for account creation in beta - https://phabricator.wikimedia.org/T87841#2991850 (10Tgr) 05Open>03Resolved [00:48:28] 10Beta-Cluster-Infrastructure: Account creation throttling too restrictive on Beta Cluster - https://phabricator.wikimedia.org/T87704#2991893 (10Tgr) [00:48:34] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Don't throttle WMF office IP(s) for account creation in beta - https://phabricator.wikimedia.org/T87841#2991892 (10Tgr) 05Resolved>03Open [00:59:26] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Don't throttle WMF office IP(s) for account creation in beta - https://phabricator.wikimedia.org/T87841#2991916 (10Tgr) [01:00:16] 06Release-Engineering-Team, 10DBA, 06Operations, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2991922 (10mmodell) incident report: https://wikitech.wikimedia.org/wiki/Incident_documentation/20170201-Phabricator [03:59:28] Yippee, build fixed! [03:59:28] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,contintLabsSlave && UbuntuTrusty build #284: 09FIXED in 2 min 28 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/284/ [04:06:36] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #284: 04FAILURE in 9 min 36 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/284/ [08:27:04] 10MediaWiki-Releasing, 10MediaWiki-Vendor: 1.27 tarball: Unnecessary library "ruflin/elastica 2.3.1" requirement - https://phabricator.wikimedia.org/T156637#2992448 (10Legoktm) >>! In T156637#2984861, @Osnard wrote: > @Legoktm Thanks for the explanation. I understand now why there is the mediawiki/vendor repo... [09:15:01] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2992491 (10MoritzMuehlenhoff) jessie-wikimedia/backports has the same priority as main, if it gets added there, it also upgrades all labs instances... [09:22:36] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2992505 (10fgiunchedi) FWIW we're using the same method via `jessie-wikimedia/experimental` on cache boxes for e.g. nginx or the kernel, this is the... [09:30:28] 10MediaWiki-Releasing, 10MediaWiki-Vendor: 1.27 tarball: Unnecessary library "ruflin/elastica 2.3.1" requirement - https://phabricator.wikimedia.org/T156637#2992527 (10Osnard) When a user downloads the extension with Special:ExtensionDistributor, shouldn't all composer defined dependencies be resolved in the r... [09:36:22] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2992548 (10hashar) ARGHGHGHG. At the risk of derailing completely: Debian.org has 500 by default and backports at 100 | codename/component | Pri... [09:38:53] 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2992552 (10MoritzMuehlenhoff) > I am not sure though why `jessie-wikimedia` has `backports` and `main` components since they end up being at the sam... [10:17:30] 06Release-Engineering-Team, 10DBA, 06Operations, 10Phabricator: Phabricator master and slave crashed - https://phabricator.wikimedia.org/T156905#2992609 (10Marostegui) Hi, Some tests I have been doing: - Executed the same query on db1048 (slave that crashed - OOM). Query took 2 minutes (as it did yester... [10:53:09] (03CR) 10Addshore: "> I am not sure how it would work. Specially the entries will be added to a more specific autoloader which is unlikely to actually be load" [integration/config] - 10https://gerrit.wikimedia.org/r/335215 (owner: 10Aleksey Bekh-Ivanov (WMDE)) [11:24:38] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2992781 (10hashar) I managed to reproduce locally with HHVM 3.12.11+dfsg-1+wmf1 from apt.wikimedia.org and its l... [12:02:41] (03CR) 10Aleksey Bekh-Ivanov (WMDE): "> I am not sure how it would work. Specially the entries will be added to a more specific autoloader which is unlikely to actually be load" [integration/config] - 10https://gerrit.wikimedia.org/r/335215 (owner: 10Aleksey Bekh-Ivanov (WMDE)) [12:25:27] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2992912 (10hashar) Bash limits the size of core files to 0 which means no core get generated. Gotta set it to `u... [12:35:04] 10Deployment-Systems, 06Release-Engineering-Team, 05Mediawiki SWAT Deployments: Clarify SWAT process for testing maintence script changes (to not use mwdebug* hosts) - https://phabricator.wikimedia.org/T153316#2876393 (10Dereckson) That means you can only use mwrepl and not mwscript eval.php on mdebug1002, s... [12:43:06] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2992971 (10hashar) Installed on mwdebug1001 under /home/hashar/mediawiki-core ``` git clone --depth 1 --single-... [12:46:12] g/win go #nasqueron [12:51:29] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2992981 (10hashar) On deployment-tin additionally did: git clone --depth 1 --single-branch https://gerrit.wikim... [12:55:37] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2992999 (10hashar) Deployment-tin does not segfault. I am thus upgrading HHVM: Unpacking hhvm-dbg (3.12.11+dfsg... [13:17:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2993037 (10hashar) Fast repro is: hhvm -v Eval.Jit=false tests/phpunit/phpunit.php tests/phpunit/includes/i... [13:22:26] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:31:30] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2990073 (10MoritzMuehlenhoff) I think I've identified the problem, a new package is building on copper ATM. [13:32:27] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [13:39:29] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2993094 (10hashar) [13:49:09] PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:50:29] (03CR) 10Addshore: "> It will work fine as soon as multiple autoloaders are allowed and just kick in aone after another (it is documented), and code in Wikiba" [integration/config] - 10https://gerrit.wikimedia.org/r/335215 (owner: 10Aleksey Bekh-Ivanov (WMDE)) [14:15:10] !log Nodepool: delete the image building of Jessie (image id 1322) to prevent a faulty HHVM version from being added. T156923 [14:15:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:15:15] T156923: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923 [14:16:09] 10scap, 15User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#2993188 (10Addshore) [14:19:01] 10scap, 06Operations, 15User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#2993201 (10Addshore) [14:25:51] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Labs instance ci-jessie-wikimedia-498353 can not be deleted - https://phabricator.wikimedia.org/T156636#2993248 (10hashar) ci-jessie-wikimedia-498353 eventually got deleted so the original issue is solved. Maybe when nova-api got restar... [14:27:06] Project beta-scap-eqiad build #140488: 04FAILURE in 1 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/140488/ [14:28:42] 10Deployment-Systems, 06Release-Engineering-Team, 10scap, 06Operations, 15User-Addshore: cannot delete non-empty directory: php-1.29.0-wmf.3 messages on 'scap sync' on mwdebug1002 - https://phabricator.wikimedia.org/T157030#2993250 (10hashar) [14:37:21] 10Deployment-Systems, 06Release-Engineering-Team, 05Mediawiki SWAT Deployments: Clarify SWAT process for testing maintence script changes (to not use mwdebug* hosts) - https://phabricator.wikimedia.org/T153316#2993262 (10hashar) Both T150912 and T153316 are attempts to use mwdebug1001 / mwdebug1002 as either... [14:37:28] Yippee, build fixed! [14:37:29] Project beta-scap-eqiad build #140489: 09FIXED in 1 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/140489/ [14:37:58] 10Deployment-Systems, 06Release-Engineering-Team, 05Mediawiki SWAT Deployments: Clarify SWAT process for testing maintence script changes (to not use mwdebug* hosts) - https://phabricator.wikimedia.org/T153316#2876393 (10hashar) [14:46:04] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#2993302 (10zeljkofilipin) Apologies for the late reply. I am looking into this. [14:53:52] twentyafterfour: is there teh ability to fire post commit hooks atm in diffusion? [14:54:35] chasemp: yes, though very manually [14:55:03] as in the initial setup of what to fire what is manual and probably in a script somewhere? [14:55:04] there is also herald which can trigger certain actions on commits and it's got a fairly powerful set of matching rules [14:55:53] afaik there is just one global script that we could hook into - phabricator already has it's own hooks and we can have phab call a custom script for every commit [14:56:01] it's not per-repo as far as I remember [14:56:14] herald is much more flexible but less powerful (the actions it can trigger are limited) [14:56:32] I'll look at the herald for diffuion magic to see if I can make sense of it [14:56:35] gotcha [14:56:47] what is it you want to trigger? [14:56:52] 06Release-Engineering-Team, 06Operations: Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org - https://phabricator.wikimedia.org/T157038#2993373 (10MoritzMuehlenhoff) [14:57:15] I'm thinking through a library distribution pipeline and wondering about automated wheel or deb building or $other [14:57:23] so idea phase [14:57:42] ahh [14:57:53] well, in that case harbormaster + herald is probably the ticket [14:58:12] herald to match the repo + commit to a build plan, then harbormaster triggers a build [14:58:29] are you guys using that for anything yet? [14:58:54] the simplest harbormaster build plan is just an http hook that sends an arbitrary http request to something like jenkins or a custom remote daemon [14:59:02] yeah we use harbormaster for scap builds [14:59:05] works really well [14:59:18] harbormaster does an http request to jenkins to fire off a build [14:59:33] is that setup in a repo somewhere I can read over? [14:59:44] it can pass along a bunch of variables like the repo url and commit hash [14:59:56] there isn't any code to read it's all configured in phab [15:00:20] understood [15:00:22] there is jenkins job builder stuff, but that's only needed if you really want to use jenkins [15:00:45] and we will hopefully be replacing that with jenkins pipelines (or something else entirely, depending on current project outcome) [15:01:05] I'm rooting for pipelines though, it looks like a nice package [15:01:30] (twss?) [15:01:33] do you remember when i converted over all the independent scripts to the pylibs repo at dA [15:01:39] that's basically the problem I want to solve [15:01:52] but distribution there was mcollective + git [15:02:09] and I think puppet + git here is widly unpopular (and for decent reasons) [15:02:19] right [15:02:41] well, we do have existing infrastructure to build debs in jenkins already [15:02:46] that's how scap gets built [15:02:50] I actually loved how pylibs turnd out there for use and distribution [15:03:03] is it tag based or? [15:03:13] but the prod package gets built by fillipo manually because jenkins can't publish to apt.wm [15:03:48] every commit of scap builds a package and pushes it to an aptly repo on beta [15:04:10] that is super interesting [15:04:24] but it could promote any commit that's tagged a certain way, for example [15:04:24] I gotta afk for a bit but I'll ping you later if I'm lost on untangling that [15:04:37] ok it's complex so I should walk you through it ;) [15:05:15] probably could do something better than the debian glue we use currently (git-buildpackage is a mess, IMO) [15:05:26] wheels would be better [15:07:44] thanks twentyafterfour ttyl [15:50:49] (03CR) 10Aleksey Bekh-Ivanov (WMDE): "> Then WilibaseLib entry point would be run and the Wikibase autoloader wouldn't be run as the other stuff has already been loaded?" [integration/config] - 10https://gerrit.wikimedia.org/r/335215 (owner: 10Aleksey Bekh-Ivanov (WMDE)) [15:59:09] (03PS1) 10Hashar: [tools-webservice] add non voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/335662 (https://phabricator.wikimedia.org/T156651) [15:59:21] I'd like to be able to create repositories in https://phabricator.wikimedia.org/diffusion/ Can someone help me with that? [15:59:42] Specifically I need to create a repo like https://phabricator.wikimedia.org/diffusion/1913/ that mirrors a github repo. [16:00:25] (03CR) 10Hashar: [C: 032] [tools-webservice] add non voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/335662 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [16:01:17] (03Merged) 10jenkins-bot: [tools-webservice] add non voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/335662 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [16:06:45] 10Continuous-Integration-Config, 06Labs, 10Tool-Labs, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2982620 (10hashar) I have added a non voting job to build the package. Unfortunately i... [16:09:40] Also, I'm looking to set up to use archiva for deploying some big assets. [16:09:56] halfek you need to be a repo admin to do that. [16:10:02] twentyafterfour ^^ [16:10:15] * halfak requests repo adminship [16:10:16] woops wrong name halfek = halfak [16:10:17] ;) [16:11:00] I mean repo admin as being able to control all repo's, though you could request edit and push requests to repo's individually i think. [16:11:17] paladox, I'm already a project admin on phab. [16:11:29] Oh yep, repo admin is different. [16:11:47] Being a leader of a production project... I need to work with diffusion a lot. I mostly try to work around it now. [16:12:24] halfak do you need the repo in gerrit and being able to view it on diffusion? [16:13:41] paladox, I need a repo on diffusion for doing deployments from. [16:13:50] oh differential [16:13:59] differential and diffusion [16:14:10] E.g. https://phabricator.wikimedia.org/diffusion/1913/ [16:14:22] halfak please see https://phabricator.wikimedia.org/project/view/85/ [16:14:39] Aha! [16:14:48] oh [16:14:51] i see [16:17:28] !log integration-slave-jessie-1001 wiping /srv/pbuilder/base-trusty-i386.cow/ it was not properly provisioned causing build to fail (eg lack of /etc/hosts) Running puppet to reprocvision it (poke T156651) [16:17:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:17:32] T156651: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651 [16:18:31] Notice: /Stage[main]/Package_builder::Environments/Package_builder::Pbuilder_base[trusty-i386]/Exec[cowbuilder_init_trusty-i386]/returns: executed successfully [16:18:34] puppet is magic sometime [16:20:57] 10Continuous-Integration-Config, 06Labs, 10Tool-Labs, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2993714 (10hashar) I fixed the instance that lacked a proper base image for Trusty. D... [16:20:57] 10Continuous-Integration-Config, 06Labs, 10Tool-Labs, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2993715 (10hashar) I fixed the instance that lacked a proper base image for Trusty. D... [16:26:03] !log deploying 9fd75a1 ores in beta [16:26:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:49:36] halfak: and it is still down :/ [16:55:54] Project beta-scap-eqiad build #140503: 04FAILURE in 0.31 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/140503/ [16:57:21] 16:55:54 16:55:54 scap failed: LockFailedError Failed to acquire lock "/var/lock/scap"; owner is "thcipriani"; reason is "test new logstash checker" (duration: 00m 00s) [16:57:52] yup, manually syncing something to test a change [16:58:06] ah ok :) [16:58:19] good looking out :) [17:07:47] Yippee, build fixed! [17:07:48] Project beta-scap-eqiad build #140504: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/140504/ [17:30:52] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations, 07HHVM: New HHVM 3.12.11 segfault at end of MediaWiki PHPUnit tests - https://phabricator.wikimedia.org/T156923#2994029 (10MoritzMuehlenhoff) I backed out the bzip2-segfault-sweep.patch introduced in 3.12.11+dfsg-1 and that... [17:33:45] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org - https://phabricator.wikimedia.org/T157038#2994039 (10Paladox) [17:50:11] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2994110 (10RobH) [18:19:52] 06Release-Engineering-Team, 10Elasticsearch, 10Phabricator (Search): Add support for elasticsearch 5 - https://phabricator.wikimedia.org/T155299#2994200 (10mmodell) [19:02:09] 10Continuous-Integration-Config, 10VisualEditor: Some CI jobs are failing because node is now on node 6 ("Assertion error: node version v6.9.1 does not match '^v4[.]'") - https://phabricator.wikimedia.org/T155685#2994399 (10Jdforrester-WMF) a:05Jdforrester-WMF>03hashar [19:15:44] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-Tobi_WMDE_SW, 15User-zeljkofilipin: Increase in failures caused by Saucelabs - https://phabricator.wikimedia.org/T152963#2994436 (10zeljkofilipin) [[ https://status.saucelabs.com/ | Sauce Labs status ]] says everything is fine on their end, there were no i... [19:17:11] Project beta-scap-eqiad build #140516: 15ABORTED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/140516/ [19:20:31] Project beta-scap-eqiad build #140517: 04FAILURE in 0.43 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/140517/ [19:27:00] ^ got the beta-scap-eqiad problem. Previous abort left lockfile in place [19:31:29] Yippee, build fixed! [19:31:29] Project beta-scap-eqiad build #140518: 09FIXED in 6 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/140518/ [19:31:53] thcipriani: ah I think I have killed the job abruptly sorry [19:32:11] thcipriani: not sure how Jenkins kills it, but most probably kill -9 which would let the lock file behind bah [19:32:27] hashar: nbd, easy to fix :) [19:34:57] Hmmm, does KeyboardInterrupt halfway through scap release the lock file? [19:35:00] * ostriches looks [19:35:31] it *should* iirc [19:35:55] well, it should even if I don't remember correctly [19:36:00] bug it *does* iirc [19:36:05] *but [19:36:45] Looks like it...I think [19:36:55] I feel like we'd hit a lot more complaints if it didn't :p [19:37:54] bah I brken jenkins again eek [19:49:40] thcipriani: what would be the recommended remedy for when git-fat fails to hydrate [19:49:43] ? [19:50:33] urandom: Bug ops about getting the new version of git-fat packaged and out ;-) [19:50:34] urandom: blerg :( if you do a git fat pull on that box as the deploy-service user that should work [19:50:39] That too ^ [19:50:41] also what ostriches said [19:51:14] thcipriani: ok [19:51:24] thcipriani: that's what i've been doing, but it feels naughty [19:51:29] urandom: rather than reimplement a piece of git-fat inside scap we opted to fix git-fat. There is a fix in our fork currently. [19:51:32] heh [19:51:37] (and i can't do it everywhere it needs doing this time around) [19:51:55] oh? [19:52:43] oh, can i sudo as deploy-service everywhere, i wonder? [19:52:54] i've been assuming that i needed to as root [19:52:55] well you should be able to do... [19:53:18] sudo -u deploy-service, as root, i mean [19:53:31] but it looks like i can, as me, on the restbase cluster at least [19:53:39] ok cool [19:53:58] I was going to suggest sshing directly since you definitely have permission to do that. [19:54:17] by "directly" I mean "as deploy-service from tin" :) [19:55:40] oh, i see [19:58:10] thcipriani: thanks for the (pending) git-fat fix, btw [19:58:35] thcipriani: it is/will be a relief not worry about this [19:58:37] urandom: yeah, definitely, it was a weird bug for sure. [19:59:11] initially I thought it had to be something wrong with git-fat, but it turns out the way git implements smudge and clean filters is a bit weird. [19:59:50] git fat just had a workaround that doesn't work for automated use. [20:00:48] (which is why a manual git fat pull always works) [20:00:55] auh, ok [20:14:44] thcipriani: i think i may have missed a step, when you said that i should be able to ssh as user deploy-service from tin [20:15:18] yeah: SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service [host] [20:15:19] thcipriani: how would that work? [20:15:37] im going to migrate db data from phab-01 to phabricator now :) [20:15:53] twentyafterfour ^^ [20:16:24] thcipriani: !? [20:16:30] thcipriani: ok, that's interesting [20:17:31] urandom: yeah, that's how scap gets to the boxes, roughly. There may be some other indirection. [20:19:01] thcipriani: Idea for scap subcommand.... `scap ssh host`? [20:19:11] To wrap the AUTH_SOCK stuff [20:20:20] paladox: what? [20:20:52] greg-g we are migrating phab-01 to phabricator (instance called that in labs) So that it is controlled by a puppet class [20:20:58] ostriches: seems reasonable. It is totally non-obvious how it works without digging in the code. [20:21:18] i've disccused this with mutante and twentyafterfour :) [20:21:36] paladox: ok, that clarification would have been nice the first time (that you're talking about 2 labs instances) [20:21:46] Oh sorry. [20:21:48] godspeed [20:39:01] wow phab-01 has 1.1 or 1.2gb's of mysql data. [20:55:18] does that include uploaded images or are they in file system [20:55:55] paladox: i just wanted to show how https://phabricator-01.wmflabs.org/ works and something happened [20:56:04] that looked a lot better 5 min ago [20:56:15] mutante yeh i am doing some mysql tweeks [20:56:18] ok :) [20:56:32] according the the issues reported here https://phabricator-01.wmflabs.org/config/issue/ [20:59:02] 10Continuous-Integration-Config, 06Labs, 10Tool-Labs, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2994645 (10scfc) >>! In T156651#2993715, @hashar wrote: > […] > Looks like it is all h... [21:01:43] (03PS1) 10Hashar: [tools-webservice] debian-glue is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/335696 (https://phabricator.wikimedia.org/T156651) [21:02:11] (03PS1) 10Hashar: [tools-manifest] add non voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/335697 (https://phabricator.wikimedia.org/T156651) [21:09:50] (03CR) 10Hashar: [C: 032] [tools-webservice] debian-glue is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/335696 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [21:09:52] (03CR) 10Hashar: [C: 032] [tools-manifest] add non voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/335697 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [21:10:51] (03Merged) 10jenkins-bot: [tools-webservice] debian-glue is now voting [integration/config] - 10https://gerrit.wikimedia.org/r/335696 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [21:10:52] (03Merged) 10jenkins-bot: [tools-manifest] add non voting debian-glue [integration/config] - 10https://gerrit.wikimedia.org/r/335697 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [21:16:17] hashar: are you going to rename the pattern for image creation then? https://gerrit.wikimedia.org/r/#/c/335373/ [21:19:56] mutante i fixed it now :) [21:19:59] should work [21:24:27] paladox: it does :) nice work! [21:24:38] yep :) [21:25:04] mutante searching wont work as i carn't overide the search config for specifing the elasticsearch version or host in phab. [21:25:13] need to find a way to support it. [21:49:27] chasemp: havent thought about it yet. One culprit is that the jessie image fails right now [21:50:02] chasemp: the hhvm version in apt.wm.o segfault on jessie. Moritz and I have been debugging it today, he rebuild a new package [21:50:08] so tomorrow I can get an image regenated [21:50:12] regenerated [21:50:24] ok no worries, I'll circle back tomorrow [21:50:25] and I guess rename them. Easier done during european mornings [21:51:00] chasemp: the script in itself is a huge thing already :] [21:53:37] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T155525#2994778 (10mmodell) [21:54:34] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T155525#2994784 (10Krinkle) [21:55:49] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T155525#2946165 (10Krinkle) [21:56:00] !log integration-slave-jessie-1001 wiping /srv/pbuilder/base-trusty-amd64.cow it was not properly provisioned causing build to fail (eg lack of /etc/hosts) Running puppet to reprocvision it (poke T156651) [21:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:56:04] T156651: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651 [21:56:26] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T155525#2946165 (10Krinkle) [22:06:53] 10Continuous-Integration-Config, 06Labs, 10Tool-Labs, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2994837 (10hashar) tools-webservice now has a the debian-glue job voting. For tools-m... [22:07:18] 10Continuous-Integration-Config, 06Labs, 10Tool-Labs, 13Patch-For-Review: operations/software/tools-webservice (and operations/software/tools-manifest?) do not run Debian tests - https://phabricator.wikimedia.org/T156651#2982620 (10hashar) p:05Triage>03Normal [22:10:53] mutante uploaded images should have been migrated too as they are in the db, none are stored on the file system :) [22:19:05] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2994879 (10RobH) [22:26:19] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2994888 (10RobH) [22:30:55] (03CR) 10GuerellaNuke23: "Why non-voting?" [integration/config] - 10https://gerrit.wikimedia.org/r/335697 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [22:31:11] (03CR) 10GuerellaNuke23: "It should probably be voting-modified" [integration/config] - 10https://gerrit.wikimedia.org/r/335697 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [22:31:42] (03CR) 10GuerellaNuke23: "zuul/layout.yaml, that is, should reflect those changes. Don't you think?" [integration/config] - 10https://gerrit.wikimedia.org/r/335697 (https://phabricator.wikimedia.org/T156651) (owner: 10Hashar) [22:31:58] (03PS1) 10GuerellaNuke23: Revert "[tools-manifest] add non voting debian-glue" [integration/config] - 10https://gerrit.wikimedia.org/r/335712 [22:32:27] 10Gerrit, 06Release-Engineering-Team, 06Operations, 13Patch-For-Review: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2994909 (10RobH) a:05RobH>03demon Assigning this task to Chad. Once he is aware that this system is all theirs, he can resolve. [22:32:35] 10Gerrit, 06Release-Engineering-Team, 06Operations: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2994911 (10RobH) [22:34:48] (03PS2) 10GuerellaNuke23: Revert "[tools-manifest] add non voting debian-glue" [integration/config] - 10https://gerrit.wikimedia.org/r/335712 [22:43:19] (03Abandoned) 10Hashar: Revert "[tools-manifest] add non voting debian-glue" [integration/config] - 10https://gerrit.wikimedia.org/r/335712 (owner: 10GuerellaNuke23) [23:04:35] paladox: got it, that explains the size of the DB i meant [23:04:45] Ok :) [23:05:05] mutante i think it's also repo's too. [23:08:02] 10Gerrit, 06Release-Engineering-Team, 06Operations: setup/install gerrit2001/WMF6408 - https://phabricator.wikimedia.org/T152525#2995039 (10Dzahn) @demon So if we'd just put the role gerrit::server on this one as well, let's figure out which things need to be stopped or skipped when not on the "active" serve... [23:08:34] paladox: here you go http://puppet-compiler.wmflabs.org/5322/ [23:08:51] mutante thanks, looks like it worked :) [23:11:03] i don't know if it does, honestly [23:11:26] looks like it's adding search config and thre was none before [23:11:38] as opposed to "make a hardcoded thing a flexible thing while not changing it" [23:12:18] but "don't know" really means don't know and not "i think it does not" [23:14:31] if it's really just making something customizable, wouldn't it be no-op ? [23:18:47] 10Deployment-Systems, 06Release-Engineering-Team, 06MediaWiki-Stakeholders-Group, 10MediaWiki-extensions-General-or-Unknown, 10Developer-Wishlist (2017): Improve LTS support of extensions - https://phabricator.wikimedia.org/T156640#2995054 (10srishakatux)