[01:11:53] !log Repool integration-slave-precise-1013 and integration-slave-trusty-1015 (live hack with libeatmydata enabled for mysql; T96308) [01:11:59] Logged the message, Master [03:19:04] So it looks to me like pybal already can react quickly to depool a server if you just simply stop apache on said server. [03:19:54] it maintains a persistent http connection to the server and depools if the connection terminates and it isn't able to reconnect immediately [03:22:45] it also supports polling a monitor url, so if we can wrangle apache to return an error state from that url ...then it'll mark the host as down. I'm not sure how long it'll take to notice the host coming back online though [03:46:24] 10Deployment-Systems, 7Epic, 3releng-201415-Q4: EPIC: The future of MediaWiki deployment: Tooling - https://phabricator.wikimedia.org/T94620#1214682 (10mmodell) @dduvall, @thcipriani, @demon: This is a fairly helpful high level overview/comparison of salt and ansible, maybe worth a read: http://jensrantil.gi... [05:08:44] 10Continuous-Integration: Merge extensions PHPUnit and QUnit jobs - https://phabricator.wikimedia.org/T88207#1214716 (10Legoktm) So AIUI we basically want to add the "qunit-karma" builder to the mediawiki-extensions-* jobs after phpunit is run? [05:23:41] (03PS1) 10Legoktm: Use generic phpunit job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/204706 (https://phabricator.wikimedia.org/T86659) [05:33:14] (03CR) 10Legoktm: [C: 032] Use generic phpunit job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/204706 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [05:35:34] (03PS1) 10Legoktm: Pin generic 'phpunit' job to labs slaves [integration/config] - 10https://gerrit.wikimedia.org/r/204707 (https://phabricator.wikimedia.org/T86659) [05:36:15] (03Merged) 10jenkins-bot: Use generic phpunit job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/204706 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [05:37:00] !log deploying https://gerrit.wikimedia.org/r/204706 [05:37:04] Logged the message, Master [05:39:10] (03CR) 10Legoktm: [C: 032] Pin generic 'phpunit' job to labs slaves [integration/config] - 10https://gerrit.wikimedia.org/r/204707 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [05:41:07] (03Merged) 10jenkins-bot: Pin generic 'phpunit' job to labs slaves [integration/config] - 10https://gerrit.wikimedia.org/r/204707 (https://phabricator.wikimedia.org/T86659) (owner: 10Legoktm) [06:36:29] 10Deployment-Systems, 6Services: Evaluate Ansible as a deployment tool - https://phabricator.wikimedia.org/T93433#1214762 (10mmodell) @GWicke: I'm totally with you on pretty much every point, though I'm not the only one that has to be convinced. [06:54:39] PROBLEM - Puppet failure on deployment-sentry2 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:19:39] RECOVERY - Puppet failure on deployment-sentry2 is OK: OK: Less than 1.00% above the threshold [0.0] [08:14:45] hashar: sorry, can we cancel the meeting today? :( [08:15:16] zeljkof: sure :D [08:15:39] zeljkof: we can do a short checkin this afternoon around 2pm if you want [08:15:44] up to you. I will be around [08:15:55] I have a meeting 2-3pm [08:16:03] I am free 3-4 [08:17:29] hashar: just take a look at my calendar and pick any time I am free :) [08:17:44] I have moved most of the meetings to friday, so it is crowded [08:19:36] 10Continuous-Integration, 7Tracking: MySQL tunning on CI slaves (tracking) - https://phabricator.wikimedia.org/T96249#1214873 (10hashar) I gave `engine=MEMORY` a quick try on https://gerrit.wikimedia.org/r/#/c/199633/ but eventually gave up because that engine does not support BLOB/TEXT columns so the install... [08:20:06] zeljkof: so lets cancel. Not sure how available I will be after 15:30 [08:20:12] zeljkof: we can catch up on monday [08:20:23] hashar: sure, sorry about the late notice [08:21:54] zeljkof: all fine. Have a lot to do anyway :) [08:22:19] hashar: two sick kids and I became an uncle today, busy morning :) [08:23:31] zeljkof: congratulations :-D [08:32:52] hashar: thanks [08:34:05] zeljkof: cestitke! [08:34:12] hashar: hvala :) [08:34:23] mobrovac: hvala :) [08:34:34] (and good luck with the sick children) [08:34:35] :) [08:34:55] mobrovac: all under control, but working from home with two sick kids... fun times :) [08:35:15] oh yeah i bet [10:51:03] 10Browser-Tests, 6Release-Engineering, 10MediaWiki-Vagrant: Vagrant command for running browser tests - https://phabricator.wikimedia.org/T96283#1214994 (10zeljkofilipin) [12:01:57] aharoni: coming? [12:12:45] !log Jenkins: enabled plugin "ZMQ Event Publisher" and publishing all jobs result on TCP port 8888 [12:12:50] Logged the message, Master [12:14:24] !log Switching Zuul scheduler on gallium.wikimedia.org to the Debian package version [12:14:26] Logged the message, Master [12:16:50] 10Continuous-Integration, 10MediaWiki-Database, 10MediaWiki-Installer: Creating MySQL tables for MediaWiki sometimes stalled on I/O for several minutes - https://phabricator.wikimedia.org/T96229#1215099 (10Krinkle) p:5Triage>3High [12:17:09] 10Continuous-Integration, 7Tracking: MySQL tunning on CI slaves (tracking) - https://phabricator.wikimedia.org/T96249#1215101 (10Krinkle) p:5Triage>3Normal [12:18:11] 10Continuous-Integration, 7Jenkins, 7Upstream: Jenkins: Builds (for beta cluster and browser tests) are stuck forever if IRC notification failed - https://phabricator.wikimedia.org/T96183#1215105 (10Krinkle) p:5Triage>3Normal [12:18:41] 10Continuous-Integration: reduce copies of mediawiki/core in workspaces - https://phabricator.wikimedia.org/T93703#1215108 (10Krinkle) p:5Triage>3Normal [12:18:55] 10Continuous-Integration, 5Continuous-Integration-Isolation: Write a migration plan for CI infra to the disposable VMs infrastructure - https://phabricator.wikimedia.org/T86172#1215110 (10Krinkle) p:5Triage>3Normal [12:25:31] PROBLEM - zuul_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-server [12:26:51] PROBLEM - zuul_merger_service_running on gallium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/local/bin/zuul-merger [12:36:57] hashar: https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/12810/console [12:37:03] 12:32:25 IOError: Lock for file '/mnt/jenkins-workspace/workspace/mediawiki-extensions-zend@2/src/extensions/cldr/.git/config' did already exist, delete '/mnt/jenkins-workspace/workspace/mediawiki-extensions-zend@2/src/extensions/cldr/.git/config.lock' in case the lock is illegal [12:37:08] I thought we fixed that? [12:37:23] * Krinkle ssh'es to fix lock [12:38:30] !log Switching zuul on lanthanum.eqiad.wmnet to the Debian package version [12:38:33] Logged the message, Master [12:40:48] 10Continuous-Integration, 7Upstream: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#1215139 (10Krinkle) Still happens. E.g. on https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/12810/console just now >12:32:25 IOError: Lock for file '/mnt/j... [12:40:49] lanthanum:/etc# zuul-cloner --version [12:40:49] Zuul version: 2.0.0-304-g685ca22-wmf1precise1 [12:41:17] gotta hard restart Jenkins again [12:41:29] Krinkle: Jenkins is once again deadlocked because of the IRC plugin [12:42:26] !log restarting Jenkins [12:42:28] That's dead lock type number 5 [12:42:29] Logged the message, Master [12:48:11] RECOVERY - zuul_merger_service_running on gallium is OK: PROCS OK: 1 process with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger [12:48:32] RECOVERY - zuul_service_running on gallium is OK: PROCS OK: 2 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-server [12:52:20] hashar: Is the package made from upstream or from our fork? [12:52:43] Does it still have this patch in it? Or something similar at least? https://review.openstack.org/#/c/147101/2 [12:53:24] PROBLEM - Host integration-slave-precise-1013 is DOWN: PING CRITICAL - Packet loss = 37%, RTA = 5103.33 ms [12:57:42] Krinkle: the .deb package ships what ever we had previously [12:57:49] I have build it using the same upstream point [12:57:58] and applied the patches we had. Nothing new has been added [12:58:19] I now have to write the step by step guide to update the .deb package [12:58:23] which is a bit nasty [12:59:26] hashar: How many local/unmerged patches do we have that are not in upstream yet? [12:59:53] 3 or 4 [13:00:20] * Ensure the repository configuration lock is released [13:00:24] * Update merge status after merge:merge is submitted [13:00:29] * Merger: ensure_cloned() now looks for '.git' [13:00:39] the patches are in a branch https://gerrit.wikimedia.org/r/#/q/status:open+project:integration/zuul+branch:patch-queue/debian/precise-wikimedia,n,z [13:01:38] Not merged in our branch? [13:02:00] the deb package uses the upstream branch point [13:02:13] and the patches are flat files in a directory debian/patches [13:02:33] I am using an utility that generates the flat files patches out of a branch patch-queue/debian/precise-wikimedia [13:02:33] https://github.com/wikimedia/integration-zuul/tree/debian/precise-wikimedia/debian/patches [13:02:36] They are merged. [13:02:38] which make it easier to rebase them [13:02:53] yeah those are the flat files [13:02:53] It looked like they weren't merged because that gerrit link shows them open? [13:03:06] and the changes are not merged indeed [13:04:20] So what are the open changes for? Or do you use them to create the flat files? [13:04:29] (so you can easily rebase and re-create the flat files next time) [13:04:32] yup [13:04:36] ok [13:04:42] so when we need to bump [13:04:56] we would checkout upstream [13:05:04] checkout whatever upstream version we want [13:05:09] then checkout debian/precise-wikimedia [13:05:13] merge in the upstream branch [13:05:26] checkout patch-queue/debian/precise-wikimedia and rebase it [13:05:37] the problem is that somehow I have merged that patch queue branch [13:05:43] gotta figure it out [13:06:04] at least now Zuul is deployed using a .deb package ! [13:06:57] May want to do rebase instead of merge when updating. so that our patches stay on top and not buried in the history. [13:08:19] Although I guess for debian package, the patches wouldn't actually be in the git history as changes to those files, they'd be in flat files. [13:08:22] Which are applied by the builder? [13:08:38] Anyway, the io lock is back. [13:08:43] 10Continuous-Integration: Upgrade Zuul server to latest upstream - https://phabricator.wikimedia.org/T94409#1215162 (10hashar) [13:08:44] 10Continuous-Integration: Zuul: python git assert error assert len(fetch_info_lines) == len(fetch_head_info) - https://phabricator.wikimedia.org/T61991#1215161 (10hashar) [13:08:46] 10Continuous-Integration: Zuul: upgrade to latest upstream version - https://phabricator.wikimedia.org/T48354#1215163 (10hashar) [13:08:48] 10Continuous-Integration, 5Continuous-Integration-Isolation, 6operations, 7Blocked-on-Operations, and 2 others: Create a Debian package for Zuul - https://phabricator.wikimedia.org/T48552#1215158 (10hashar) 5Open>3Resolved We now have Zuul packages for our Precise and Trusty distributions thanks to @fg... [13:11:21] The stack trace says it's coming from this line https://phabricator.wikimedia.org/T86730#1215139 https://github.com/wikimedia/integration-zuul/blob/master/zuul/merger/merger.py#L58 [13:24:09] Krinkle: arf my fault [13:24:18] I forgot to pick the patches from https://phabricator.wikimedia.org/T86730 [13:24:28] guess I have to solve that patch management thing now [13:42:19] 10Deployment-Systems: [Trebuchet] git fetch writes corrupted objects when its connection to the web service is lost during fetch - https://phabricator.wikimedia.org/T53142#1215296 (10Aklapper) Proposing to close this task as no reply to greg's comment above? And should this really still be assigned to rlane32? [13:42:33] (03PS4) 10Hashar: Replace python shebang with python2.7 [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195540 [13:42:35] (03PS4) 10Hashar: Merger: ensure_cloned() now looks for '.git' [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195281 [13:42:37] (03PS4) 10Hashar: wmf: soften requirements [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195280 [13:42:39] (03PS4) 10Hashar: Ensure the repository configuration lock is released [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195283 [13:42:41] (03PS4) 10Hashar: Update merge status after merge:merge is submitted [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195282 [13:43:32] (03Abandoned) 10Hashar: Package python deps with dh-virtualenv [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/195541 (https://phabricator.wikimedia.org/T48552) (owner: 10Hashar) [13:49:34] Krinkle: my bad. Actually ori patch to release the git lock is in the deb package [13:49:48] Krinkle: but it does not solve the problem of lock being left behind when a job is killled [13:51:03] hashar: Yeah, because config_writer also needs a llck [13:51:08] which is not inside the try catch [13:51:25] only config_write.write() is [13:51:27] git.repo._release() might do the trick [13:51:36] https://review.openstack.org/#/c/147101/2 [13:52:56] see inline comments [13:53:44] hashar: This problem goes away if we wipe workspace and clone from local git cache. [13:54:16] Well need that anyway in isolation, we could work on that before that to make sure it works properly [13:54:21] and also solve these kinds of problems. [13:55:17] and add --depth=10 or something like that to all clone actions zuul-cloner does. [13:55:39] since there's no need for full clones, even with hardlinks, we don't need the entire repo [13:55:54] Though if we have hard links, we can do it without trouble [13:56:07] hashar: I guess the main blocker for git cache is a way to keep it up to date? [13:56:21] without race condition / corruption when a slave is being used. [13:57:21] It will solve the lock problem, the mediawiki-core workspace copy problem, and the scaling problem (we can enable l10n bot because workspaces would be cheap) [14:01:11] I am not sure about the git cache and the corruptions [14:01:16] have to look at it seriously [14:01:20] the lock problem [14:01:34] a poor man solution would be to add a find command that deletes them before we invoke zuul cloner [14:01:36] something like [14:01:57] find src/ -wholename '*/.git/*lock' -delete [14:03:08] Should we do that? [14:04:06] that is reasonably easy [14:04:15] doesn't need us to figure out the python hack that would release-) [14:04:18] so yeah [14:04:24] that is lame but should do the job for now [14:05:37] 10Continuous-Integration, 10Wikimedia-Hackathon-2015, 7Upstream: All new extensions should be setup automatically with Zuul - https://phabricator.wikimedia.org/T92909#1215336 (10Krinkle) There is no good way to know which entry points a repository needs/wants: npm, composer, publish (jsduck, doxygen, coverag... [14:05:47] 10Continuous-Integration: Upgrade Zuul server to latest upstream - https://phabricator.wikimedia.org/T94409#1215340 (10Krinkle) [14:05:48] 10Continuous-Integration: Zuul should support wildcard projects - https://phabricator.wikimedia.org/T76256#793854 (10Krinkle) [14:05:50] 10Continuous-Integration, 10Wikimedia-Hackathon-2015, 7Upstream: All new extensions should be setup automatically with Zuul - https://phabricator.wikimedia.org/T92909#1215338 (10Krinkle) [14:05:56] 10Continuous-Integration, 10Wikimedia-Hackathon-2015: All new extensions should be setup automatically with Zuul - https://phabricator.wikimedia.org/T92909#1123551 (10Krinkle) [14:06:34] 10Continuous-Integration, 7Upstream: Zuul should support wildcard projects - https://phabricator.wikimedia.org/T76256#793854 (10Krinkle) [14:06:36] manybubbles: you are cutting out [14:09:18] 10Continuous-Integration, 7Upstream: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#1215358 (10hashar) The patch at https://review.openstack.org/#/c/147101/ handle the config.lock case. It would not release other locks that might be caused when Zuul abrupt... [14:09:25] Krinkle: I have commented on it [14:09:31] gotta leave, be back later tonight [14:09:34] kid & wife sicks [14:09:39] and they woke up from nap [14:11:27] 10Continuous-Integration, 7Upstream: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#1215360 (10Krinkle) >>! In T86730#1215358, @hashar wrote: > The patch at https://review.openstack.org/#/c/147101/ handle the config.lock case. It would not release other lo... [14:43:38] 10Continuous-Integration, 5Patch-For-Review: Switch MySQL storage to tmpfs - https://phabricator.wikimedia.org/T96230#1215473 (10Krinkle) This was rolled out between 17:20 and 18:00 on 2014-04-16. I've took samples from jobs for MediaWiki core master and wmf branches (e.g. REL1_23 is not comparable). I also ex... [15:28:11] (03PS1) 10Krinkle: Use DOC_SUBPATH instead of ZUUL_BRANCH for misc publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/204782 [15:30:25] (03CR) 10jenkins-bot: [V: 04-1] Use DOC_SUBPATH instead of ZUUL_BRANCH for misc publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/204782 (owner: 10Krinkle) [15:31:31] (03PS1) 10Krinkle: Add unicodejs-publish to publish pipeline (handles git tags) [integration/config] - 10https://gerrit.wikimedia.org/r/204783 [15:33:22] (03CR) 10jenkins-bot: [V: 04-1] Add unicodejs-publish to publish pipeline (handles git tags) [integration/config] - 10https://gerrit.wikimedia.org/r/204783 (owner: 10Krinkle) [15:33:48] (03PS1) 10Krinkle: Update zuul-server path, now at /usr/bin/zuul-server [integration/config] - 10https://gerrit.wikimedia.org/r/204785 [15:35:49] (03CR) 10jenkins-bot: [V: 04-1] Update zuul-server path, now at /usr/bin/zuul-server [integration/config] - 10https://gerrit.wikimedia.org/r/204785 (owner: 10Krinkle) [15:37:16] (03CR) 10Krinkle: "Deployed. Chicken-egg problem. Needs to approve itself." [integration/config] - 10https://gerrit.wikimedia.org/r/204785 (owner: 10Krinkle) [15:37:22] (03CR) 10Krinkle: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/204785 (owner: 10Krinkle) [15:37:26] (03CR) 10Krinkle: [C: 032] Update zuul-server path, now at /usr/bin/zuul-server [integration/config] - 10https://gerrit.wikimedia.org/r/204785 (owner: 10Krinkle) [15:39:22] (03Merged) 10jenkins-bot: Update zuul-server path, now at /usr/bin/zuul-server [integration/config] - 10https://gerrit.wikimedia.org/r/204785 (owner: 10Krinkle) [15:39:55] (03PS2) 10Krinkle: Use DOC_SUBPATH instead of ZUUL_BRANCH for misc publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/204782 [15:42:44] (03CR) 10Krinkle: [C: 032] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/204782 (owner: 10Krinkle) [15:44:40] (03Merged) 10jenkins-bot: Use DOC_SUBPATH instead of ZUUL_BRANCH for misc publish jobs [integration/config] - 10https://gerrit.wikimedia.org/r/204782 (owner: 10Krinkle) [15:48:18] 10Deployment-Systems: [Trebuchet] git fetch writes corrupted objects when its connection to the web service is lost during fetch - https://phabricator.wikimedia.org/T53142#1215633 (10greg) a:5RyanLane>3None [15:48:27] (03PS2) 10Krinkle: Add unicodejs-publish to publish pipeline (handles git tags) [integration/config] - 10https://gerrit.wikimedia.org/r/204783 [15:49:00] 10Continuous-Integration, 5Patch-For-Review: Switch MySQL storage to tmpfs - https://phabricator.wikimedia.org/T96230#1215635 (10Krinkle) 5Open>3Resolved [15:49:01] 10Continuous-Integration, 7Tracking: MySQL tunning on CI slaves (tracking) - https://phabricator.wikimedia.org/T96249#1215636 (10Krinkle) [15:49:17] 10Deployment-Systems, 7Epic, 3releng-201415-Q4: EPIC: The future of MediaWiki deployment: Tooling - https://phabricator.wikimedia.org/T94620#1215639 (10greg) btw, see also: {T45338} [15:50:37] 10Continuous-Integration, 10MediaWiki-Unit-tests, 7JavaScript: Apache on Jenkins slave takes over 30s to respond (QUnit/AJAX "Test timed out") - https://phabricator.wikimedia.org/T95971#1215651 (10Krinkle) 5Open>3Resolved Haven't seen these time outs for at least 24 hours. Re-open if it comes back but fo... [15:53:26] 10Continuous-Integration, 10MediaWiki-Database, 10MediaWiki-Installer: Creating MySQL tables for MediaWiki sometimes stalled on I/O for several minutes - https://phabricator.wikimedia.org/T96229#1215662 (10Krinkle) 5Open>3Resolved a:3Krinkle I haven't seen any jobs lock for more than 1 minute since T96... [15:55:25] (03CR) 10Krinkle: [C: 032] "This is probably not enough to make it work (due to unresolved blockers at T73062), but adding the pipeline doesn't hurt. For testing purp" [integration/config] - 10https://gerrit.wikimedia.org/r/204783 (owner: 10Krinkle) [15:57:33] (03Merged) 10jenkins-bot: Add unicodejs-publish to publish pipeline (handles git tags) [integration/config] - 10https://gerrit.wikimedia.org/r/204783 (owner: 10Krinkle) [16:00:01] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/204783 [16:00:06] Logged the message, Master [16:02:41] Yippee, build fixed! [16:02:42] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce build #220: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-10-sauce/220/ [16:05:04] I wonder why that test was failing and now it's not ^ ? Anyone have any ideas [16:05:29] I'm able to log in to SauceLabs now but I don't know if I can access screenshots of tests run by Jenkins... [16:12:01] (03PS1) 10Krinkle: Split up misc *-publish from *-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/204791 [16:12:46] (03PS2) 10Krinkle: Split up misc *-publish from *-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/204791 (https://phabricator.wikimedia.org/T73062) [16:25:14] (03CR) 10Krinkle: [C: 032] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/204791 (https://phabricator.wikimedia.org/T73062) (owner: 10Krinkle) [16:27:27] (03Merged) 10jenkins-bot: Split up misc *-publish from *-coverage [integration/config] - 10https://gerrit.wikimedia.org/r/204791 (https://phabricator.wikimedia.org/T73062) (owner: 10Krinkle) [16:29:03] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/204791 [16:29:06] Logged the message, Master [16:32:26] Yippee, build fixed! [16:32:27] Project browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce build #42: FIXED in 1 min 34 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.m.wikipedia.beta.wmflabs.org-os_x_10.10-iphone-sauce/42/ [16:36:54] 6Release-Engineering, 10MediaWiki-Maintenance-scripts, 10MediaWiki-Redirects, 5Patch-For-Review: namespaceDupes not handling deleted namespace redirects as desired - https://phabricator.wikimedia.org/T91401#1215811 (10demon) Logging table is all cleaned up. Any other weirdness people are spotting? [16:41:54] 6Release-Engineering, 10MediaWiki-Maintenance-scripts, 10MediaWiki-Redirects, 5Patch-For-Review: namespaceDupes not handling deleted namespace redirects as desired - https://phabricator.wikimedia.org/T91401#1215828 (10demon) p:5Unbreak!>3High [16:50:47] 10Continuous-Integration, 5Patch-For-Review: Switch MySQL storage to tmpfs - https://phabricator.wikimedia.org/T96230#1215905 (10hashar) Excellent! I love the arrows on the build time graphs. [16:53:05] 10Continuous-Integration, 7Upstream: Allow ref-updated listener to filter out tag deletions - https://phabricator.wikimedia.org/T96390#1215921 (10Krinkle) 3NEW [17:02:10] ^d: thcipriani : in case you missed it from last night: depooling with pybal will be straightforward - we just shut down apache and that takes care of it as far as I can tell... [17:03:41] <^d> That's kind of missing the point. We want to explicitly depool it, not have pybal depool because it's down [17:03:52] <^d> (Which would result in icinga warning that would have to be ack'd, etc) [17:05:22] I did miss that. I think moving towards a lock-file-based solution seems like the most elegant way to handle depooling. Have no idea _how_ that would work at this point, but it feels like a desirable outcome. [17:08:08] <^d> Also, what if we want to bring apache back up to test the host, but don't want it pooled in case it's still busted? [17:08:17] <^d> Yes, I think explicit depooling is best [17:08:37] then we need to add an interface to pybal [17:09:11] <^d> Agreed [17:09:34] the existing pybal monitoring mechanism seems like the most straight forward way ... if we could have a specialized return status that causes the monitoring to depool but not alert [17:10:02] (03PS1) 10Krinkle: Don't run mediawiki-core-doxygen-publish on wmf tags [integration/config] - 10https://gerrit.wikimedia.org/r/204812 [17:10:28] (03PS2) 10Krinkle: Don't run mediawiki-core-doxygen-publish on wmf tags [integration/config] - 10https://gerrit.wikimedia.org/r/204812 (https://phabricator.wikimedia.org/T73062) [17:10:35] (03CR) 10Krinkle: [C: 032] Don't run mediawiki-core-doxygen-publish on wmf tags [integration/config] - 10https://gerrit.wikimedia.org/r/204812 (https://phabricator.wikimedia.org/T73062) (owner: 10Krinkle) [17:11:24] it would be cleaner IMO if the "lock file" or whatever flag we use, is on the target host, and if pybal is polling that host, then it'd be easier for pybal to discover the "depool flag" rather than have pybal listen to a network interface with a rest api or whatever [17:12:40] <^d> Makes sense [17:12:44] I don't know how pybal is configured specifically - do we have it polling a status url or just using the persistent http connection? [17:13:14] <^d> It polls based on commands we give it in puppet [17:13:19] it polls every 10 seconds, which is often enough to react quickly for deployments (I think that's fast enough anyway) [17:13:35] commands in puppet? [17:13:47] * twentyafterfour searches ops/puppet [17:13:50] <^d> Sec, I'm grepping for context [17:14:27] <^d> Grep for 'runcommand' [17:14:56] <^d> check-apache & co [17:16:03] <^d> lvs::configuration is really where all the magic comes together [17:16:53] <^d> So yeah, adding some lock file to the host and having check-apache check that (or adding a 2nd monitor) should do it [17:18:00] # Open an SSH connection to the real-server. The command is overridden by the authorized_keys file. [17:18:10] what's in the authorized keys file? [17:18:40] <^d> heck if I know :) [17:19:01] I'll ass #operations [17:19:04] ask [17:19:07] lol [17:19:15] let me ass you a question [17:19:38] it looks like just the key [17:19:43] so not sure what that comment means [17:20:02] (I'm noisy and following along) [17:22:40] command="uptime; touch /var/tmp/pybal-check.stamp" on deployment [17:22:55] ^ twentyafterfour [17:23:29] thcipriani: thanks [17:23:38] chasemp: it uses "forced command" [17:23:48] maybe encoded? I don't know [17:26:02] so as long as the ssh command returns 0 pybal is ok with it. it probably doesn't capture stdout. So we would have a difficult time making that check do something special I suppose [17:26:21] <^d> We could easily add a 2nd check [17:26:32] <^d> Although 2 connections sucks, nvm [17:27:01] maybe I can make bypal check stdout or figure out a way to make ssh return a specific error code? [17:27:23] I don't know why we are using ssh for monitoring honestly, seems strange. For web servers we should be monitoring http(s) [17:27:35] twentyafterfour: is the key concated with the command param in a template? must be [17:27:46] could you do something simple like: test ! -f /some/file [17:27:53] but yeah forced command is just saying "user can only do this" [17:28:00] thcipriani: yes [17:28:23] chasemp: pybal does ssh into the web server with no command specified, and relys in forced command to specify the command [17:28:43] understood yes [17:28:48] thcipriani: if it returns an error status though we are back to "how do we not throw alerts" [17:29:26] we need a way to return a _special_ status that bypasses alerting but still does the depooling [17:30:34] for web servers we could monitor http which would hit a script on the web server that checks the file and returns a different http status code in that case. but apparently our pybal config doesn't monitor http? [17:30:49] trigger some third script that checks for file, if exists depools, in not no-op, returns 0. Which sounds like a horrendous idea :) [17:31:18] also, we must be missing some other check pybal does, right? [17:31:34] pybal supports multiple methods of checking the target server [17:31:49] but I don't know which ones are in use... it's configurable. looks like we just use ssh :-/ [17:33:28] part of the downside of the check do now has to be it has no load / response time context? [17:34:23] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216089 (10GWicke) [17:34:29] 'ProxyFetch' => { [17:34:32] 'url' => [ 'http://en.wikipedia.org/wiki/Main_Page' ], [17:34:34] }, [17:34:50] so it does fetch an http url but that's to the proxy not the individual nodes [17:35:46] chasemp: yeah it seems like the load distribution is manually calculated, then entered into puppet as static configuration values [17:35:54] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:36:46] (03CR) 10Krinkle: [C: 032] Don't run mediawiki-core-doxygen-publish on wmf tags [integration/config] - 10https://gerrit.wikimedia.org/r/204812 (https://phabricator.wikimedia.org/T73062) (owner: 10Krinkle) [17:37:00] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216101 (10GWicke) [17:38:07] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:41:04] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216124 (10GWicke) [17:41:39] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:42:19] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:43:25] test -f /etc/pybal/depool && exit 1 || uptime; touch /var/tmp/pybal-check.stamp [17:43:28] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216135 (10GWicke) [17:43:35] I guess it fairly sane [17:43:39] is [17:43:59] chasemp: that makes sense but we need to make pybal differentiate between that and a failed server [17:44:00] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:44:25] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:44:26] I don't know if it currently differentiates between ssh connection failing vs. returning a specific code. [17:44:40] yeah you are stuck translating exit codes in taht case as is [17:44:40] (03Merged) 10jenkins-bot: Don't run mediawiki-core-doxygen-publish on wmf tags [integration/config] - 10https://gerrit.wikimedia.org/r/204812 (https://phabricator.wikimedia.org/T73062) (owner: 10Krinkle) [17:44:59] twentyafterfour: probably not but then again, failed ssh should probably be depool? even though that's kinda dangerous [17:45:16] !log Creating integration-slave-trusty-1017 [17:45:19] Logged the message, Master [17:45:19] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1136887 (10GWicke) [17:45:24] yes but we want a depool without alerting [17:45:35] chasemp: failed ssh would depool and alert [17:45:38] I've never seen a depool alert [17:45:45] not saying they don't exist but [17:45:49] oh? I'm just basing this one what ^d said [17:45:49] what do they look like? [17:46:06] I don't knwo [17:46:15] <^d> Not a depool alert [17:46:24] <^d> But an alert from apache being down on the host [17:46:32] <^d> (the original idea: just kill apache) [17:46:47] well ssh returns the exit code, so we could exit 123; and have pybal check the specific exit code [17:46:47] ah ok so yeah shouldn't be an issue if you are [17:46:56] just failing the check intentionally [17:47:21] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216155 (10GWicke) [17:47:22] well then, this is probably the solution ... [17:47:26] in all seriousness though if you guys are into this [17:47:35] please make a "percentage of hosts depooled" check [17:47:39] so we don't all die in a fire :) [17:48:00] as I think pool / depool now does rely on apache check since they are synonymous [17:48:01] essentially [17:48:28] yeah, we should check what the pybal ssh command is in prod [17:49:33] best practice would be for there to be a pool / depool check in icinga and have the deploy silence the check as it goes [17:49:34] thcipriani: I bet it's the same as deployment. /src/ops/puppet/modules/mediawiki/files/pybal_key: [17:49:34] as it depools [17:49:42] it's 'uptime; touch /var/tmp/pybal-check.stamp' [17:49:42] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216163 (10GWicke) [17:49:43] in prod [17:50:04] chasemp: how do we silence a icinga check programmatically ? [17:50:31] at the moment it sucks but we could deploy https://github.com/zorkian/nagios-api [17:50:37] it works shockingly well, ended up using it at dA [17:50:41] for all the maintenance scripts [17:51:32] but yeah scary if nothing programmatic cares that a mw* is depooled [17:52:19] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/204812 [17:52:21] Logged the message, Master [17:53:05] 6Release-Engineering: Convert old wmf/* deployment branches to tags - https://phabricator.wikimedia.org/T1288#1216174 (10Krinkle) p:5Triage>3Normal [18:25:54] 10Deployment-Systems, 6Release-Engineering, 6Services, 6operations: Streamline our service development and deployment process - https://phabricator.wikimedia.org/T93428#1216329 (10GWicke) [18:26:13] 6Release-Engineering, 10MediaWiki-Maintenance-scripts, 10MediaWiki-Redirects, 5Patch-For-Review: namespaceDupes not handling deleted namespace redirects as desired - https://phabricator.wikimedia.org/T91401#1216330 (10He7d3r) Yeah, we are still waiting for the redlinks to existing pages be purged. E.g.: ht... [18:31:38] 10Browser-Tests, 6Release-Engineering, 10VisualEditor: Selenium bug with Firefox causes VE test failure - https://phabricator.wikimedia.org/T90651#1216342 (10Aklapper) 5Open>3Resolved a:3Aklapper >>! In T90651#1157077, @zeljkofilipin wrote: > I see the commit is merged. Can this be closed? No reply. A... [18:37:31] 10Beta-Cluster: Occasionally getting 403 HTTP Method not allowed from bits - https://phabricator.wikimedia.org/T93021#1216384 (10Aklapper) Still an issue? [19:11:30] 6Release-Engineering, 10MediaWiki-Maintenance-scripts, 10MediaWiki-Redirects, 5Patch-For-Review: namespaceDupes not handling deleted namespace redirects as desired - https://phabricator.wikimedia.org/T91401#1216557 (10demon) Running `refreshLinks` for all pages on ptwiki. [19:14:17] (03PS1) 10Krinkle: Enable footer by default in IntegrationPage.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/204847 [19:14:19] (03PS1) 10Krinkle: Page: Move title handling from newDirIndex() to handleDirIndex() [integration/docroot] - 10https://gerrit.wikimedia.org/r/204848 [19:14:44] (03CR) 10Krinkle: [C: 032] Enable footer by default in IntegrationPage.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/204847 (owner: 10Krinkle) [19:14:47] (03Merged) 10jenkins-bot: Enable footer by default in IntegrationPage.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/204847 (owner: 10Krinkle) [19:15:25] (03CR) 10Krinkle: [C: 032] Page: Move title handling from newDirIndex() to handleDirIndex() [integration/docroot] - 10https://gerrit.wikimedia.org/r/204848 (owner: 10Krinkle) [19:15:28] (03Merged) 10jenkins-bot: Page: Move title handling from newDirIndex() to handleDirIndex() [integration/docroot] - 10https://gerrit.wikimedia.org/r/204848 (owner: 10Krinkle) [19:15:57] 10Beta-Cluster, 6operations: Beta Cluster no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#1216564 (10RobH) Shouldn't the option of : a) Have an Opsen populate /etc/ssl/private/star.wmflabs.org.key on all of the frontend boxes for beta [0]. This private key must match the public key in... [19:17:57] 10Deployment-Systems, 6Services, 6operations: Automate compiling service dependencies using production Jessie libraries - https://phabricator.wikimedia.org/T94611#1168065 (10mobrovac) A PR promoting the usage of a local Docker container to build the proper dependencies and set up the deploy repo can be found... [19:18:43] 10Beta-Cluster, 6operations: Beta Cluster no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#1216577 (10RobH) I also think suggesting that ops isn't willing to help is both counter-productive, and incorrect. I was pinged on this today by someone not even on the task, and this doesn't have... [19:21:17] 10Beta-Cluster, 6operations: Beta Cluster no longer listens for HTTPS - https://phabricator.wikimedia.org/T70387#1216582 (10Dzahn) duplicate of T50501 ? [19:22:09] (03PS1) 10Krinkle: Remove redundant RewriteBase rule [integration/docroot] - 10https://gerrit.wikimedia.org/r/204853 [19:22:11] (03PS1) 10Krinkle: doc: Enable pretty directory index for doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/204854 [19:22:17] 10Beta-Cluster, 10Wikimedia-Labs-Infrastructure, 6operations: beta: Get SSL certificates for *.{projects}.beta.wmflabs.org - https://phabricator.wikimedia.org/T50501#526960 (10Dzahn) Is T70387 a duplicate of this? [19:22:21] (03CR) 10Krinkle: [C: 032] Remove redundant RewriteBase rule [integration/docroot] - 10https://gerrit.wikimedia.org/r/204853 (owner: 10Krinkle) [19:22:23] (03Merged) 10jenkins-bot: Remove redundant RewriteBase rule [integration/docroot] - 10https://gerrit.wikimedia.org/r/204853 (owner: 10Krinkle) [19:22:41] (03CR) 10Krinkle: [C: 032] doc: Enable pretty directory index for doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/204854 (owner: 10Krinkle) [19:22:43] (03Merged) 10jenkins-bot: doc: Enable pretty directory index for doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/204854 (owner: 10Krinkle) [19:31:23] 10Continuous-Integration, 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review, 7Upstream: Create a Debian package for NodePool on Debian Jessie - https://phabricator.wikimedia.org/T89142#1216616 (10hashar) During our Apr 17th checkin, we pointed out a new .deb package targeting Jessie shoul... [19:42:56] 5Continuous-Integration-Isolation, 6operations: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216685 (10hashar) 3NEW [19:43:09] 6Release-Engineering, 10Ops-Access-Requests, 6Phabricator, 6operations: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1216691 (10chasemp) [19:44:08] 6Release-Engineering, 10Ops-Access-Requests, 6Phabricator, 6operations: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1216702 (10chasemp) p:5Triage>3Normal [19:44:25] 6Release-Engineering, 10Ops-Access-Requests, 6Phabricator, 6operations: Change twentyafterfour and demon to root on phabricator (iridium) - https://phabricator.wikimedia.org/T96425#1216676 (10chasemp) @greg can you stamp this with your undying love and approval? [19:46:57] 5Continuous-Integration-Isolation, 6operations: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216708 (10Dzahn) a:3Dzahn [20:00:39] 5Continuous-Integration-Isolation, 6operations: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216754 (10hashar) There is a puppet change to have ferm rules use hiera instead of networks.pp: //contint: move zuul_merger_hosts to hi... [20:03:27] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #624: FAILURE in 16 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/624/ [20:10:57] 10Beta-Cluster, 10VisualEditor, 10Wikimedia-Search: File search on beta labs returns results from production commons and beta commons. VE search results are in unexpected order also. - https://phabricator.wikimedia.org/T90650#1216804 (10Jdforrester-WMF) 5Open>3declined [20:11:48] 5Continuous-Integration-Isolation, 6operations: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216809 (10Dzahn) a:5Dzahn>3None [20:17:38] greg: the wmf-deployments calendar on google is wrong. can you give me access to update it? [20:24:59] 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216857 (10hashar) Patch is stalled pending discussion about the behavior of puppet @resolve(()) if it can't DNS res... [20:28:14] 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216867 (10hashar) a:3hashar [20:28:44] 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review: Allow gearman and zeromq connections from labnodepool1001 to gallium.wikimedia.org - https://phabricator.wikimedia.org/T96426#1216685 (10hashar) p:5Normal>3High [20:35:12] 10Continuous-Integration, 5Continuous-Integration-Isolation, 6operations: Provide Debian package python-pymysql for jessie-wikimedia - https://phabricator.wikimedia.org/T96131#1216901 (10hashar) a:3Andrew Following up Apr 17th checkin, @Andrew kindly accepted to look at the package :) As to why upstream i... [20:35:31] 10Continuous-Integration, 5Continuous-Integration-Isolation, 6operations: Provide Debian package python-pymysql for jessie-wikimedia - https://phabricator.wikimedia.org/T96131#1216904 (10hashar) p:5Normal>3Low The task is not blocking the implementation of the service since I have manually installed the... [20:36:57] 10Continuous-Integration, 6Release-Engineering: Run qunit tests in IE8 (and possibly other class A browsers) - https://phabricator.wikimedia.org/T96432#1216912 (10EBernhardson) 3NEW [20:45:40] hashar: I don't want to poke you in the eye on language but on https://phabricator.wikimedia.org/T96131#1216904 [20:45:52] definitely can't consider it implemented until that is all coming from puppet etc [20:46:00] but unsure what you mean by implemented :) [20:50:24] 10Continuous-Integration, 6Release-Engineering: Run qunit tests in IE8 (and possibly other class A browsers) - https://phabricator.wikimedia.org/T96432#1216988 (10EBernhardson) [20:50:34] <^d> thcipriani: I went ahead and merged your checkoutMW patch, that part works fine [20:50:48] <^d> Still some breakage on the edgecases with updateBranchPointers, but that can wait for next week :) [20:50:51] ^d: nice! Thanks [20:51:00] sounds like fun [20:51:11] <^d> Also, I think YuviPanda and I got a working patch for mediawiki-installation scap targets [20:51:31] I saw that, inline template-ish thing [20:51:48] It is still -2d from joe tho [20:51:59] seems like the only way to customize it without +2 on ops/puppet :\ [20:52:01] 5Continuous-Integration-Isolation, 6operations: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1216990 (10hashar) Following Friday 2015-04-10 checkin, @chasemp talked to @mark about the labs VLANs. During today checkin, that is stalled pending further discussion... [20:52:08] <^d> YuviPanda: I think your version is nicer than mine though :) [20:52:13] <^d> At least you avoid that template [20:52:55] Yeah [20:52:59] The template was the only thing I didn't like [20:53:17] <^d> Actually I think we can do the same with scap_proxies too [20:53:23] <^d> And kill the template [20:55:09] Yeah [20:55:13] <^d> Parsoid too [20:55:19] <^d> Really the whole thing could be generic [20:55:39] <^d> You pass a $title for the filename and the variable to lookup entries from [20:56:03] chasemp: let me grab an english/french dictionary and figure out what "to implement" means :) [20:56:14] hah ok [20:56:32] I was referring to the first phase where I am basically hacking stuff as root [20:56:36] no worries all I wanted to say was, we gotta do that before we put the thing in service [20:56:42] ok understood [20:56:45] while documenting my action in uncommited puppet [20:56:52] not sure which verb would work [20:57:01] willing to learn! [20:57:31] ^d: there is a create_resources function in puppet, but that way lies madness [20:58:21] I actually started to use it in a patch for mediawiki-installations, then realized what I was doing was horribly insecure :P [20:58:28] <^d> Actually, I think I got it [20:59:18] chasemp: rephrased. Thanks for the suggestion [21:01:10] I like create_resources() myself though there is not that many use cases for it [21:01:44] ^d: one example is manifests/role/protoproxy.pp: create_resources( protoproxy, $instances, $defaults ) [21:01:51] was meant to install nginx on the varnish caches [21:04:39] makes it easy to invoke the same class over and over https://gerrit.wikimedia.org/r/#/c/63644/1/manifests/role/protoproxy.pp,unified [21:51:17] 10Continuous-Integration, 5Continuous-Integration-Isolation, 6operations, 5Patch-For-Review, 7Upstream: Create a Debian package for NodePool on Debian Jessie - https://phabricator.wikimedia.org/T89142#1217217 (10Andrew) [21:51:19] 10Continuous-Integration, 5Continuous-Integration-Isolation, 6operations: Provide Debian package python-pymysql for jessie-wikimedia - https://phabricator.wikimedia.org/T96131#1217215 (10Andrew) 5Open>3Resolved done! [22:14:41] 10Browser-Tests, 10MediaWiki-extensions-ZeroBanner, 6Zero, 7Tracking: [Blocking] Chromium/Firefox do not provide ability to set custom headers via Selenium and PhantomJS does not clear session (LocalStorage/Cookies/Cache) properly - https://phabricator.wikimedia.org/T96451#1217298 (10jhobs) 3NEW [22:15:01] 10Browser-Tests, 10MediaWiki-extensions-ZeroBanner, 6Zero, 7Tracking: [Blocking] Chromium/Firefox do not provide ability to set custom headers via Selenium and PhantomJS does not clear session (LocalStorage/Cookies/Cache) properly - https://phabricator.wikimedia.org/T96451#1217305 (10jhobs) [22:15:36] 10Browser-Tests, 10MediaWiki-extensions-ZeroBanner, 6Zero, 7Tracking: [Blocking] Chromium/Firefox do not provide ability to set custom headers via Selenium and PhantomJS does not clear session (LocalStorage/Cookies/Cache) properly - https://phabricator.wikimedia.org/T96451#1217298 (10jhobs) [22:53:52] 6Release-Engineering, 3Team-Practices-This-Week: Test phabricator sprint extension updates - https://phabricator.wikimedia.org/T95469#1217373 (10chasemp) >>! In T95469#1199733, @ksmith wrote: > Assigning to Kristen, per discussions in the TPG weekly meeting where we concluded that she is uniquely qualified to... [23:07:01] 6Release-Engineering, 3Team-Practices-This-Week: Test phabricator sprint extension updates - https://phabricator.wikimedia.org/T95469#1217398 (10Awjrichards) @Klans_WMF ^^