[04:18:00] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #77: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/77/ [05:51:28] Project mediawiki-core-code-coverage build #2142: 04STILL FAILING in 2 hr 51 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2142/ [08:11:31] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10DBA, 10Datasets-General-or-Unknown, and 2 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2469832 (10ArielGlenn) [08:32:06] !log gallium: upgrading Zuul 2.1.0-95-g66c8e52-wmf1precise1 .. zuul_2.1.0-151-g30a433b-wmf3precise1 [08:33:17] hashar hi, and :) [08:34:09] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/299191 (https://phabricator.wikimedia.org/T140499) (owner: 10Paladox) [08:35:06] (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/299191 (https://phabricator.wikimedia.org/T140499) (owner: 10Paladox) [08:35:40] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2469944 (10hashar) [08:35:42] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Zuul-cloner failing to acquire .git lock sometimes - https://phabricator.wikimedia.org/T86730#2469945 (10hashar) [08:35:44] 10Continuous-Integration-Infrastructure, 10Zuul, 13Patch-For-Review: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2469941 (10hashar) 05Open>03Resolved Fixed by moving an import ( https://gerrit.wikimedia.org/r/299136 ) Pushed / inclu... [08:39:34] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2469953 (10hashar) 05Open>03Resolved I have deployed Zuul at 8:35am UTC with version 2.1.0-151-g30a433b-wmf3precise1 . The connection issue with Gerrit (T137525) has... [08:43:42] hashar do the graphs work [08:43:42] ? [08:49:27] We can test multi connections on the labs instance i setup with gerrit [08:49:35] having both gerrit 2.12 and 2.8 [08:49:36] :) [08:49:51] need to install gerrit 2.8 though. [08:49:56] hashar ^^ [08:51:25] We will need to update the jessie zuul package so that i know i did everything correctly [09:06:18] :) [09:43:23] Yay graphs work [11:32:34] * paladox has setup gerrit 2.8 on the gerrit-test instance to test zuul with mutiple connections on gerrit 2.12 and 2.8 [11:34:18] * paladox http://gerrit-old.wmflabs.org [11:39:30] Project beta-code-update-eqiad build #113382: 04FAILURE in 6 min 28 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/113382/ [11:41:55] hashar ive setup gerrit 2.8 [11:42:03] http://gerrit-old.wmflabs.org [11:42:28] so now we can test zuul mutiple connections with gerrit 2.12 and gerrit 2.8 on the gerrit-test instance. [11:49:28] Project beta-code-update-eqiad build #113383: 04STILL FAILING in 6 min 27 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/113383/ [11:51:31] 11:43:02 INFO:mwextpull:cwd: /srv/mediawiki-staging/php-master/extensions [11:51:31] 11:43:02 INFO:mwextpull:running: git pull [11:51:31] 11:48:03 error: RPC failed; result=22, HTTP code = 503 [11:51:31] 11:48:03 fatal: The remote end hung up unexpectedly [11:56:13] hashar could i have help with setting up ssh please. [11:56:34] Reedy: Gerrit is borked somehow https://phabricator.wikimedia.org/T140604 [11:56:45] aha [11:59:28] Project beta-code-update-eqiad build #113384: 04STILL FAILING in 6 min 27 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/113384/ [12:04:31] Yippee, build fixed! [12:04:32] Project beta-code-update-eqiad build #113385: 09FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/113385/ [12:04:49] :) [12:07:24] Reedy: Gerrit is back [12:07:36] hashar: Hence the :) above ;) [12:08:48] !log Enabling puppet again on integration-slave-precise-1002 , removing Zuul-server config and adding the slave back in Jenkins pool [12:08:54] pff [12:10:00] qa-morebots: [12:10:00] I am a logbot running on tools-exec-1220. [12:10:00] Messages are logged to https://tools.wmflabs.org/sal/releng. [12:10:00] To log a message, type !log . [12:10:02] !log Enabling puppet again on integration-slave-precise-1002 , removing Zuul-server config and adding the slave back in Jenkins pool [12:10:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:10:09] !log (restarted qa-morebots) [12:10:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:13:59] It wont start the ssh 29419 on gerrit 2.8 [12:24:41] I got ssh to work [12:40:47] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10DBA, 10Datasets-General-or-Unknown, and 2 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2470388 (10ArielGlenn) Couple more followup chang... [13:29:59] hashar ive updated the zuul conf files with [connection name] [13:30:04] for mutiple connections [13:44:36] twentyafterfour: just fyi (awhile yet) SSL WARNING - Certificate *.wmfusercontent.org valid until 2016-09-12 13:41:12 +0000 (expires in 55 days) [13:44:45] phab.wmfusercontent.org etc ^ [13:47:46] Yippee, build fixed! [13:47:46] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #84: 09FIXED in 3 min 45 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/84/ [14:14:33] hashar there is a bug in the connections, please could you backport https://review.openstack.org/#/c/300924/ [14:14:42] as im hitting the bug when i am testing it. [14:15:19] paladox: I will cherry pick a wild range of them eventually [14:15:27] and maybe bump Zuul to the latest version :) [14:15:49] paladox: that is when validating the layout isn't it ? [14:16:16] Oh [14:16:18] yes [14:16:29] I added two connections types [14:16:33] one called oldgerrit [14:16:37] and one called gerrit [14:16:46] py", line 443, in _parseConfig [14:16:46] params) [14:16:47] File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 383, in _getReporterDriver [14:16:48] return self._getDriver('reporter', connection_name, driver_config) [14:16:50] File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/scheduler.py", line 368, in _getDriver [14:16:53] driver = drivers[dtype][driver_name].split(':') [14:16:55] KeyError: 'oldgerrit' [14:16:57] hashar ^^ [14:16:58] ;D [14:17:09] :) [14:17:23] paladox: that is unrelated [14:17:29] https://www.mail-archive.com/openstack-infra@lists.openstack.org/msg04083.html [14:17:38] Actually that is it failing it [14:17:42] according to https://www.mail-archive.com/openstack-infra@lists.openstack.org/msg04083.html [14:17:49] which is a bug [14:18:00] apparently when parsing the config file, it can not find a connection named "oldgerrit" [14:18:05] Nope [14:18:11] that is what the KeyError: 'oldgerrit' is about [14:18:11] but i defined it in the zuul conf files [14:18:18] then maybe they are only populated when registering ? [14:18:25] Oh [14:18:34] But the above person also had the same problem [14:18:38] hitting the same error as me [14:18:55] ah neat ;] [14:19:01] Yep [14:19:12] Im going to manually add it to the jessie and repackage [14:19:14] and re test [14:19:14] now [14:19:38] hashar also there's an offtopic channel now, offical one #wikimedia-offtopic [14:19:39] :) [14:20:38] what is that channel ? [14:20:45] It is an offtopic channel [14:20:53] as not to distrupt other channels [14:20:57] what folks are talking bout ? Random crap? [14:21:02] Yes [14:21:19] Not many are in the channel currently [14:21:26] since it was only setup yesturday. [14:21:27] :) [14:22:03] (03PS1) 10Hashar: Register connections when testing configuration [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299542 [14:22:07] paladox: ^^ [14:22:11] :) [14:22:12] thanks [14:23:15] But there isent alot of random talk [14:23:51] But on friday when you left, the news came out with breaking news about turkey and so everyone was talking about it. [14:23:55] hashar ^^ [14:26:34] (03PS1) 10Hashar: 2.1.0-151-g30a433b-wmf4precise [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299545 [14:27:13] * paladox has now done applying the patch for jessie, now i am rebuilding it as a dpkg :) [14:27:16] hashar ^^ [14:27:32] (03CR) 10Paladox: [C: 031] ":)" [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299542 (owner: 10Hashar) [14:28:13] (03CR) 10Hashar: [C: 032 V: 032] Register connections when testing configuration [integration/zuul] (patch-queue/debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299542 (owner: 10Hashar) [14:29:42] 10Continuous-Integration-Config, 10MediaWiki-Debian: Set up CI auto-building for mediawiki/debian repository - https://phabricator.wikimedia.org/T122978#2470890 (10hashar) The CI permanents slaves (either Trusty or Jessie or maybe both) already have cowbuilder roots set up via the Puppet module package_builder :) [14:34:08] hashar it is built now [14:34:11] im applying it [14:36:14] Oh that didnt work [14:36:55] hashar ^^ [14:37:02] not sure why it is failing since i applied the patch [14:38:04] (03PS2) 10Hashar: 2.1.0-151-g30a433b-wmf4precise [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299545 [14:38:33] (03CR) 10Hashar: "Bumped GitPython 2.0.6 to 2.0.7" [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299545 (owner: 10Hashar) [14:38:54] paladox: no clue :) [14:38:57] Oh [14:39:12] paladox: do you have a zuul.conf example that fails and should succeed with the patch applied ? [14:39:15] I can try on my machine [14:39:25] I have it set like [14:39:25] https://gerrit.wikimedia.org/r/299542 [14:39:28] Woops [14:39:32] [connection gerrit] [14:39:32] driver=gerrit [14:39:32] server=127.0.0.1 [14:39:32] port=29418 [14:39:32] baseurl=http://gerrit-test.wmflabs.org/gerrit [14:39:33] user=jenkins [14:39:35] sshkey=/var/lib/zuul/.ssh/id_rsa [14:39:45] [connection oldgerrit] [14:39:45] driver=gerrit [14:39:45] server=127.0.0.1 [14:39:45] port=29419 [14:39:47] baseurl=http://gerrit-old.wmflabs.org/gerrit [14:39:49] user=jenkins [14:39:51] sshkey=/var/lib/zuul/.ssh/id_rsa [14:40:18] Then in layout.yam file i just copied and pasted the gerrit bit that defines the gerrit connection and just named it oldgerrit [14:46:39] paladox: paste it to https://phabricator.wikimedia.org/paste/ !! [14:46:47] Ok [14:46:52] :D [14:47:45] paladox: with a basic layout.yaml :) [14:47:55] Ok [14:48:05] I reused the one we use for production [14:48:22] and left gerrit: alone but added oldgerrit: ontop [14:48:28] but didnt add the email filters to it [14:48:31] hashar ^^ [14:49:25] https://phabricator.wikimedia.org/P3461 [14:50:22] hashar ^^ [14:51:50] I have it on gerrit-test /etc/zuul/wikimedia/zuul/layout.yaml [14:52:02] paladox: patch works for me [14:52:08] Oh [14:52:18] Maybe i did something wrong [14:55:04] I only left the new connection in test pipeline [14:55:10] paladox: replied with my tests https://phabricator.wikimedia.org/P3461#16600 [14:55:13] seems i get different error now [14:55:18] the patch you proposed definitely fixed it [14:56:00] Oh, thanks [14:56:01] :) [14:56:08] Im going throught the file now [14:56:38] return base_validate(path, iteritems(data), out) [14:56:38] File "/usr/lib/python2.7/dist-packages/voluptuous.py", line 298, in validate_mapping [14:56:38] raise MultipleInvalid(errors) [14:56:38] voluptuous.MultipleInvalid: expected a dictionary for dictionary value @ data['pipelines'][0]['success'] [14:56:43] hashar ^^ i get that now [14:57:25] that is something else missing in one of the pipeline definition I guess [14:57:46] Oh [15:00:02] hashar oh so you carn't have the same pipeline [15:00:13] !log Upgraded Zuul on the Precise slaves to zuul_2.1.0-151-g30a433b-wmf4precise1 [15:00:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [15:00:18] for the connections [15:01:30] hashar how do we reuse the same pipeline for mutiple connections [15:01:32] please [15:04:16] I have no idea [15:04:28] paladox: I have never looked at the multiple connections stuff :D [15:04:34] Oh [15:04:39] beside knowing that zuul.conf is back compatible [15:04:43] Oh [15:04:55] eg you can keep the gerrit configuration in a [gerrit] section and dont have to rename it to [connection gerrit] [15:04:59] https://phabricator.wikimedia.org/diffusion/CIZU/change/master/tests/fixtures/layouts/good_connections1.yaml;352264b3c22b501ae53c1643e31df2ccdfc89388 [15:05:04] Oh [15:05:34] hashar ^^ [15:06:03] Im guessing we can trigger from both gerrit's [15:07:35] paladox: I dont think we would do that [15:07:36] maybe [15:07:41] but probably not [15:07:42] Oh [15:07:49] So two pipelines [15:07:53] I will try that then [15:07:58] we will just switch the existing one from ytterbium to lead (the new server) [15:08:05] Ok [15:08:06] :) [15:08:14] and if we ever get a second zuul setup would have that 2nd setup to point to lead directly [15:08:22] Oh :) [15:09:29] paladox: prod updated with the patch :] [15:09:34] Yay :) [15:09:39] in case we need multiple connections [15:09:39] thanks [15:09:43] Oh :) [15:09:56] It is really hot outside :) [15:09:58] sunny too. [15:10:10] almost 30 degreess tomarror is for even hotter. [15:10:11] :) [15:10:13] hashar ^^ [15:25:10] hashar maybe i didnt apply the patch [15:25:19] but when i built it, it didnt fail [15:25:27] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, and 3 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2471189 (10thcipriani) >>! In T129151#2466150, @elukey wrote: > Suggestion: might be good to have a reference in https://doc.wikimedia.org/mw-tools-... [15:31:19] Oh wait [15:31:23] the patch didnt apply [15:31:29] I wonder what happended [15:31:32] (03PS10) 10Hashar: Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [15:32:05] 10Continuous-Integration-Config, 10Zuul, 13Patch-For-Review, 07WorkType-Maintenance: Restore integration/config Zuul version requirement - https://phabricator.wikimedia.org/T136610#2471218 (10hashar) a:03Paladox [15:32:17] paladox: I probably got your Friday patch fixed https://gerrit.wikimedia.org/r/#/c/299194/ [15:32:31] paladox: give it a look / try it :] It passes on my machine and I think it is good enough [15:32:41] Yay [15:32:45] thanks [15:32:52] (03PS11) 10Hashar: Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [15:32:53] rebasing and merging [15:32:58] Ok [15:32:59] thanks [15:33:00] :) [15:33:11] im going to rebuild the dpkg with the patch. [15:33:11] paladox: thank you to have caught that one!! [15:33:17] Your welcome :) [15:33:54] paladox: for patches I am using git buildpackage "pq" command http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/man.gbp.pq.html [15:33:56] I guess the roads will be melting tomarror due to heatwave :) [15:33:56] basically [15:33:58] Oh [15:34:24] you get the upstream source code + the cherry picks you want in a branch named: patch-queue/debian/precise-wikimedia [15:34:29] Oh [15:34:41] which is associated with the branch debian/precise-wikimedia [15:34:50] that debian/precise-wikimedia branch is based on the upstream code [15:34:54] Oh [15:35:02] Im building against jessie [15:35:10] with all changes in it being in /debian/ (no change in upstream code) [15:35:19] i merged in the debian-precise branch a week ago. [15:35:20] so the idea is you cherry pick the patches in the patch-queue branch [15:35:24] then you can do: gbp pq export [15:35:31] that will do the diff between the branches [15:35:47] and add each of the commits as a patch under debian/patches/ [15:35:53] Oh [15:35:54] that is a bit complicated though :( [15:35:59] so in short [15:36:02] Oh [15:36:25] debian/precise-wikimedia : fork of an upstream commit with changes only altering /debian/ [15:36:47] Yep [15:37:03] patch-queue/debian/precise-wikimedia : for of an upstream commit + cherry picks from upstream but NOT changing /debian/ (the directory /debian/ is not even present in that branch) [15:37:11] Oh [15:37:25] But is that branch only precise specific [15:37:31] or can all distro use it [15:37:32] I cant remember whether I wrote a documentation about that [15:37:41] so yeah for each distro you can get a pair of branches [15:37:59] debian/ and patch-queue/debian/ [15:38:05] Oh [15:38:07] (03CR) 10Hashar: [C: 032] Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [15:38:11] 10Continuous-Integration-Config, 10Zuul, 13Patch-For-Review, 07WorkType-Maintenance: Restore integration/config Zuul version requirement - https://phabricator.wikimedia.org/T136610#2471256 (10hashar) 05Open>03Resolved [15:38:12] yay ^^ :) [15:38:34] what becomes a mess is when you want to maintain multiple branches together :( [15:38:48] Oh [15:38:56] They should invent easy packaging [15:38:59] like windows does [15:39:02] so I first work on debian/precise-wikimedia [15:39:06] which is what is used on gallium [15:39:07] Yippee, build fixed! [15:39:07] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #83: 09FIXED in 17 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/83/ [15:39:08] there is this open source installer and it is really easy [15:39:11] Oh [15:39:28] then I will merge that branch in each of the debian/trusty-wikimedia and debian/jessie-wikimedia [15:39:29] I guess we can now update precise to furthur upstream [15:39:31] then backport it [15:39:34] :) [15:39:43] then tweak the debian/control for the dependencies / fix conflicts etc [15:39:47] Oh [15:39:48] that is really very very very messy [15:39:51] Yep [15:39:58] I cant even properly review my own changes ! [15:40:02] With the installer i use on windows you jsut include it your self [15:40:09] making it less messy and easy to install [15:40:13] https://integration.wikimedia.org/ci/job/integration-zuul-layoutvalidation-gate/2326/console [15:40:14] :( [15:40:18] Oh [15:40:23] (03CR) 10jenkins-bot: [V: 04-1] Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [15:40:39] 15:38:48 voluptuous.MultipleInvalid: extra keys not allowed @ data['pipelines'][0]['success']['gerrit'] [15:40:43] not sure what that is [15:41:00] 10Continuous-Integration-Config, 10Zuul, 13Patch-For-Review, 07WorkType-Maintenance: Restore integration/config Zuul version requirement - https://phabricator.wikimedia.org/T136610#2471260 (10hashar) 05Resolved>03Open Fails gate-and-submit: ``` + python /srv/deployment/integration/slave-scripts/bin/jen... [15:46:10] hashar how do i apply the patch onto of the changes i made [15:46:14] since it seems to fail dpkg [15:46:21] due to me having commit changes yet i have [15:46:24] git add -A --all [15:46:25] git commit [15:46:38] please [15:48:04] paladox: have to get away from keyboard for a bit sorry :( [15:48:09] Ok [15:50:25] greg-g: y'all have a meeting in 10, right? Can you carve out 5 minutes of that for me to ask some questions about CI? [15:51:35] andrewbogott: not at the beginning, but after Rachel is done talking offsite stuff [15:51:51] great, thanks [15:54:40] 10Continuous-Integration-Config, 10MediaWiki-extensions-MultimediaViewer: Drop old "jshint" job from MediaViewer extension test/etc. pipelines - https://phabricator.wikimedia.org/T140652#2471308 (10Jdforrester-WMF) [15:57:02] (03CR) 10Paladox: [C: 031] 2.1.0-151-g30a433b-wmf4precise [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299545 (owner: 10Hashar) [16:27:56] ok, labvirt1012,labvirt1013 are now in the scheduling pool. Nodepool should start using them pretty much right away [16:28:14] :-} [16:28:31] andrewbogott: both labvirt had instances booting on them already havent they ? [16:29:00] hashar_: yeah, the video-to-commons project has been running on 1012 and 1013 for about a week [16:29:25] and some toollabs stuff [16:29:37] andrewbogott: sounds like there is high confidence they are working fine :) [16:29:57] yeah, it's pretty low-risk [16:31:14] as the instances are being consuming you will see some new coming on the new labvirt [16:32:23] andrewbogott: and regarding metrics driven capacity planning. Yeah definitely need one, though most metrics have been reporting for just a few weeks thx to a new Zuul version [16:32:43] andrewbogott: we have talk about it to figure out how to best represent the service [16:33:16] yeah, it's not 100% obvious what metrics to care about [16:37:45] andrewbogott: we need a basic one to start with and see whether it is a good usage to track the size of the pool [16:37:49] then iterate [16:37:55] sorry that is really trial & errors process :( [16:38:10] It's the only way :) [16:44:37] andrewbogott: will be the task https://phabricator.wikimedia.org/T139771 [16:45:03] which is more or less a meta task for us to setup more KPI [16:45:24] that looks great, thank you [16:57:07] andrewbogott: and I have mailed our mailing list so we dont forget about it :) [16:59:08] 06Release-Engineering-Team, 06Operations, 10Traffic, 05Security: Make sure we're not relying on HTTP_PROXY headers - https://phabricator.wikimedia.org/T140658#2471564 (10demon) [17:12:19] 10Continuous-Integration-Config, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog: Drop old "jshint" job from MediaViewer extension test/etc. pipelines - https://phabricator.wikimedia.org/T140652#2471629 (10jhobs) [17:12:26] 10Continuous-Integration-Config, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog: Drop old "jshint" job from MediaViewer extension test/etc. pipelines - https://phabricator.wikimedia.org/T140652#2471308 (10bmansurov) Why not fix errors? [17:14:37] 10Continuous-Integration-Config, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog: Drop old "jshint" job from MediaViewer extension test/etc. pipelines - https://phabricator.wikimedia.org/T140652#2471646 (10jhobs) p:05Triage>03Low [17:19:53] 10Continuous-Integration-Config, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog: Drop old "jshint" job from MediaViewer extension test/etc. pipelines - https://phabricator.wikimedia.org/T140652#2471687 (10Jdforrester-WMF) >>! In T140652#2471632, @bmansurov wrote: > Why not fix errors? They a... [17:22:48] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2471712 (10mmodell) Maybe we could create a new priority value. Some... [17:27:06] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2471731 (10Jdforrester-WMF) >>! In T140207#2471712, @mmodell wrote: >... [17:30:44] 06Release-Engineering-Team, 15User-greg: Send vendor feedback forms to HR - https://phabricator.wikimedia.org/T140662#2471745 (10greg) [17:31:16] 06Release-Engineering-Team, 15User-greg: share casual feedback with vendors - https://phabricator.wikimedia.org/T140508#2471759 (10greg) [17:31:17] 06Release-Engineering-Team, 15User-greg: Send vendor feedback forms to HR - https://phabricator.wikimedia.org/T140662#2471758 (10greg) [17:31:30] 06Release-Engineering-Team, 15User-greg: share casual feedback with vendors - https://phabricator.wikimedia.org/T140508#2467002 (10greg) 05Open>03Resolved [17:31:32] 06Release-Engineering-Team, 15User-greg: Send vendor feedback forms to HR - https://phabricator.wikimedia.org/T140662#2471745 (10greg) [17:32:22] 06Release-Engineering-Team, 15User-greg: share casual feedback with vendors - https://phabricator.wikimedia.org/T140508#2467002 (10greg) [17:32:23] 06Release-Engineering-Team, 15User-greg: Write casual feedback for RelEng vendors - https://phabricator.wikimedia.org/T139208#2471764 (10greg) [17:34:12] 05Gerrit-Migration, 03releng-201516-q4: Phase 1 repository migrations to Differential (goal - end of June 2016) - https://phabricator.wikimedia.org/T130418#2471787 (10greg) a:05greg>03mmodell [17:35:09] 10Continuous-Integration-Infrastructure, 06Project-Admins, 10Zuul, 07Jenkins: Create tag for zuul-cloner - https://phabricator.wikimedia.org/T84945#2471790 (10greg) a:05greg>03None [17:37:18] hashar yay i got it to work now [17:37:53] I used the patch you generated [17:37:55] thanks :) [17:45:50] It is working with it validating against layout.yaml [17:46:03] even with it using mutiple connections in one pipeline [17:46:11] let's see if it will work [17:49:24] paladox: progress!! [17:55:50] hashar yep [17:56:05] hashar im now seeing if i can test with both [17:56:12] seems the gerrit-test site works [17:56:17] whereas gerrit-old dosent [17:56:38] not sure if that might be because it wont connect through ssh [18:04:26] hashar im not sure but ssh isent being picked up from gerrit-old for zuul [18:07:29] ostriches, i will put css in different patch. [18:07:38] but the thing you pointed out with /r/p [18:07:43] is the correct link [18:08:34] I see your response. [18:08:38] Ok [18:09:21] And then the other bit I mentioned about it not needing to be configurable? [18:09:38] Yep i will remove that bit too [18:09:54] okie dokie, thx [18:10:01] Thanks for explaning:) [18:10:03] :) [18:13:38] ostriches done. [18:14:01] :) [18:20:23] 05Gerrit-Migration, 10Differential, 13Patch-For-Review: Set differential.always-allow-close to true - https://phabricator.wikimedia.org/T131623#2471940 (10demon) If this is declined the associated Gerrit change should probably be abandoned too. [18:27:17] Project mediawiki-core-code-coverage build #2143: 04STILL FAILING in 3 hr 27 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2143/ [18:29:39] ssh works just doint know why it isent being picked up by zuul [18:29:41] hashar ^^ [18:33:01] https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/69154/console seems unstable - python "stderr: 'fatal: HEAD not found below refs/heads!'" [18:33:05] https://gerrit.wikimedia.org/r/#/c/299589/ [18:43:46] hashar yay it works [18:44:02] Oh wait no only in a seperate pipeline [18:44:29] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2472015 (10mmodell) >>! In T140207#2471731, @Jdforrester-WMF wrote: >... [18:46:37] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2472019 (10greg) I'm still trying to figure out how to use that proje... [18:56:13] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Convert old wmf/* deployment branches to tags (recurring chore) - https://phabricator.wikimedia.org/T1288#2472069 (10demon) We should also prune old `wmf-config/ExtensionMessages-*.php` files too. [18:56:15] 10Deployment-Systems: Make make-wmf-branch able to branch extensions with replaced substring of the version of mediawiki being branched - https://phabricator.wikimedia.org/T51392#2472070 (10mmodell) [18:56:17] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches), 03releng-201617-q1, 07Epic: Merge to deployed branches instead of cutting a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#2472071 (10mmodell) [18:56:44] 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches), 03releng-201617-q1, 07Epic: Merge to deployed branches instead of cutting a new deployment branch every week. - https://phabricator.wikimedia.org/T89945#1049281 (10mmodell) [18:56:47] 10Deployment-Systems: Make make-wmf-branch able to branch extensions with replaced substring of the version of mediawiki being branched - https://phabricator.wikimedia.org/T51392#578600 (10mmodell) [18:58:46] 10Deployment-Systems: Make make-wmf-branch able to branch extensions with replaced substring of the version of mediawiki being branched - https://phabricator.wikimedia.org/T51392#2472096 (10mmodell) Due to the current plan to migrate to long-lived-branches ({T89945}) I think that `make-wmf-branch` should be cons... [19:02:46] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2472121 (10Jdforrester-WMF) >>! In T140207#2472019, @greg wrote: > An... [19:06:30] 10Deployment-Systems, 10MediaWiki-Configuration, 05MW-1.27-release-notes, 13Patch-For-Review: extension-list should live in the mediawiki branch rather than mediawiki-config - https://phabricator.wikimedia.org/T125678#2472146 (10mmodell) @hashar is right, this isn't necessary if mergeMessageFileList.php ca... [19:53:31] 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator, 06Team-Practices: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2472503 (10Danny_B) A view from outside sharing experiences from my p... [19:59:17] hashar seems that you carn't have two connections in same pipeline [19:59:26] but it would be really good if upstream supported that. [20:42:44] (03PS12) 10Hashar: Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [20:43:17] paladox: I took over your patch! [20:43:21] hashar thanks [20:43:26] paladox: had to change the Jenkins job as well [20:43:29] hashar I got connections working. [20:43:37] but not in same pipelin [20:43:40] pipeline yet [20:43:42] :0 [20:44:10] :) [20:44:28] https://gerrit.wikimedia.org/r/#/c/299194/ SUCCESS !!! [20:44:49] now I dont even understand why it works [20:45:06] oh the job only runs on +2 [20:45:10] (03CR) 10Hashar: [C: 032] Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [20:45:19] hashar :) [20:45:24] :) [20:45:25] thanks [20:45:30] it is going to fail [20:45:35] I will refresh the job after that [20:45:45] Oh [20:45:48] Why does it fails [20:45:49] ? [20:45:58] I mean why will it fail? [20:46:05] (03CR) 10jenkins-bot: [V: 04-1] Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [20:46:08] Oh [20:46:41] 20:45:30 voluptuous.MultipleInvalid: extra keys not allowed @ data['pipelines'][0]['success']['gerrit'] [20:46:47] https://integration.wikimedia.org/ci/job/integration-zuul-layoutvalidation-gate/2327/console [20:47:52] (03PS13) 10Hashar: Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [20:48:17] 10Beta-Cluster-Infrastructure: deployment-fluorine becomes unresponsive frequently - https://phabricator.wikimedia.org/T140313#2459953 (10chasemp) this happend again today but afa nova is concerned things are ok. tyler also noted he could connect to 22 but not ssh. I'm beginning to suspect something with the h... [20:49:35] (03CR) 10Hashar: [C: 032] Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [20:52:02] (03Merged) 10jenkins-bot: Fix zuul being pinned to a specific commit [integration/config] - 10https://gerrit.wikimedia.org/r/299194 (https://phabricator.wikimedia.org/T136610) (owner: 10Paladox) [20:52:03] hashar :) [20:52:12] it worked, thanks :) :) [20:52:53] 10Beta-Cluster-Infrastructure: deployment-fluorine becomes unresponsive frequently - https://phabricator.wikimedia.org/T140313#2459953 (10thcipriani) Hmmm, well one thing is `/srv/` is full: ``` thcipriani@deployment-fluorine:/srv/mw-log$ df -h Filesystem Size Used Avail Use% Mounted o... [20:53:13] paladox: \o/ [20:53:16] 10Continuous-Integration-Config, 10Zuul, 13Patch-For-Review, 07WorkType-Maintenance: Restore integration/config Zuul version requirement - https://phabricator.wikimedia.org/T136610#2472868 (10hashar) 05Open>03Resolved Restored and even corrected to use the branch that has our cherry picks ( patch-queue... [20:53:55] hashar :) [20:55:03] 10Continuous-Integration-Config, 10Zuul, 13Patch-For-Review, 07WorkType-Maintenance: Restore integration/config Zuul version requirement - https://phabricator.wikimedia.org/T136610#2472870 (10Paladox) :) :). [20:56:42] !log Deleted deployment-fluorine:/srv/mw-log/archive/*-201605* freed 30 GB [20:56:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:58:00] * paladox wonders if there will be any more upates from upstream and merged into the debian/* branches by @hashar [20:58:02] :) [20:59:23] :) [21:00:17] For zuul [21:05:27] 10Beta-Cluster-Infrastructure: deployment-fluorine becomes unresponsive frequently - https://phabricator.wikimedia.org/T140313#2472886 (10thcipriani) Deleted `deployment-fluorine:/srv/mw-log/archive/*-20160{5,6}*` freed 30 GB. So far I can logout and still log back in since the last reboot, so there's something. [21:08:59] thcipriani: looks like we need a more strict log retention for deployment-fluorine :( [21:09:13] thcipriani: and with logstash, we can probably drop the log files very early [21:11:31] hashar did you hear about turkey? [21:12:11] paladox: yeah been watching the events live on friday night [21:12:20] Oh, i was too [21:12:26] havent looked at the aftermath or all the geopolitical implications though [21:12:32] hrm, I think this is the thin line standing between us and a full /srv/ https://github.com/wikimedia/operations-puppet/blob/production/files/misc/scripts/mw-log-cleanup [21:13:05] Oh, another user brought it to my attention about turkey so i imediatly went onto bbc news and skys news and found turkey. [21:13:10] thcipriani: eeeeek [21:13:15] heh [21:13:17] Wikipedia english was also really quick to pick it up [21:13:21] but not wikinews. [21:13:43] got to be more logrotate stuffs somewhere... [21:15:17] hashar but the funny part was that most channels on irc would have been distracted by it [21:15:22] for example #windows was. [21:15:45] (03CR) 10Hashar: [C: 032 V: 032] "Deployed on gallium and the Precise slaves." [integration/zuul] (debian/precise-wikimedia) - 10https://gerrit.wikimedia.org/r/299545 (owner: 10Hashar) [21:15:46] Where ones users asked can anyone in turkey update to the latest windows 10 build. [21:17:58] I am off / ehading to bed! [21:18:07] have happy hacking and files cleanup duties! [21:31:24] PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree (<100.00%) [21:31:48] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:32:34] PROBLEM - Puppet run on deployment-apertium01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:33:06] PROBLEM - SSH on integration-slave-trusty-1018 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:25] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:22] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:40] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:34:48] PROBLEM - Free space - all mounts on deployment-sentry2 is CRITICAL: CRITICAL: deployment-prep.deployment-sentry2.diskspace._var.byte_percentfree (<100.00%) [21:35:05] It seems some are failing ^^ [21:35:43] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:45] PROBLEM - SSH on integration-aptly01 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:35:50] PROBLEM - Puppet staleness on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [21:36:04] PROBLEM - Puppet run on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:36:13] legoktm jzerebecki ^^ [21:36:23] thcipriani ^^ [22:03:09] (03PS1) 10Mholloway: Use OpenJdk 8 for Android periodic tests [integration/config] - 10https://gerrit.wikimedia.org/r/299674 (https://phabricator.wikimedia.org/T139137) [22:15:02] 06Release-Engineering-Team, 06Project-Admins, 15User-greg: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#2473193 (10Danny_B) [22:32:00] awight|brb: When you get a chance, thoughts on https://gerrit.wikimedia.org/r/#/c/209856/ would be nice (more backlog--) [22:50:03] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: puppet failed on deployment-logstash2.deployment-prep - https://phabricator.wikimedia.org/T139368#2473392 (10EBernhardson) 05Open>03Resolved a:03EBernhardson server upgraded, puppet re-enabled. [22:53:24] PROBLEM - Host deployment-logstash3 is DOWN: CRITICAL - Host Unreachable (10.68.21.33) [22:55:46] RECOVERY - Puppet staleness on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [3600.0] [22:56:43] 06Release-Engineering-Team, 06Developer-Relations (Jul-Sep-2016), 15User-greg: Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727#2473430 (10greg) I basically just edited the etherpad to included @Aklapper's suggestions. There's also now a section on mil... [23:42:06] 10Continuous-Integration-Config, 10MediaWiki-extensions-Insider: Insider extension code C+2s not triggering jenkins - https://phabricator.wikimedia.org/T140702#2473555 (10Jdforrester-WMF)