[00:08:54] Yippee, build fixed! [00:08:54] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #204: FIXED in 24 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/204/ [04:10:52] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #273: FAILURE in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/273/ [04:46:56] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #105: FAILURE in 9 min 26 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/105/ [05:06:52] Yippee, build fixed! [05:06:52] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #230: FIXED in 1 hr 5 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/230/ [05:33:58] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #183: FAILURE in 1 min 48 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/183/ [07:31:04] good morning [07:31:09] zeljkof hello:) [07:31:14] guten morgen [07:31:22] dobro jutro :) [07:31:26] hashar: ^ [07:31:37] :D [07:33:17] Tobi_WMDE_SWE: have you solved your python 2.7.2 / JJB issue yesterday? [07:34:12] hashar: updating to 2.7.8 solved it. I did not downgrade again.. [07:34:24] \O/ [07:35:04] zeljkof: I chatted with Dan yesterday evening and he largely improved the mediawiki_selenium patch to tweak headless settings https://gerrit.wikimedia.org/r/#/c/164704/ [07:35:16] he even proposed some patches to upstream [07:37:09] (03CR) 10Hashar: [C: 031] "Excellent, that address all the concerns I had :-] Well done Dan." [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) (owner: 10Dduvall) [07:42:38] hashar: the remaining issue with the wikidata browsertest jobs is that I need to define some extra environment variables [07:42:57] Tobi_WMDE_SWE: such as ? :-D [07:43:10] can you link the change again Bitte? [07:45:05] hashar: we need separate users for a wikidata client and repo [07:45:18] so we don't just have one mediawiki username [07:45:22] doh [07:45:32] and they all run in the same job right? [07:45:41] but I can probably solve it with the config file mechanism we have [07:45:58] if you look at https://gerrit.wikimedia.org/r/#/c/163129/4/browsertests.yaml [07:46:06] line 580 [07:46:54] we have a config.yaml where we can define settings that are exported as environment variables at runtime [07:49:12] hashar: that leaves BROWSER_TIMEOUT [07:49:24] I think we can support that for all jobs [07:49:35] as it is handled in the mediawiki_selenium gem [07:49:47] only wikidata uses it at the moment AFAIK [07:57:40] back around sorry [07:57:45] being processing email Tobi_WMDE_SWE sorry [07:58:20] ohh [07:58:46] Tobi_WMDE_SWE: that might be cucumber looking for a config/config.yml file indeed. Zeljkof would know more [07:59:36] hashar: think I need to make a separate job-template for wikidata jobs though. [07:59:47] hashar, Tobi_WMDE_SWE: there is a ruby tool for that https://bugzilla.wikimedia.org/show_bug.cgi?id=69405 [08:00:00] it is on my todo list to implement it for us [08:01:18] leaving it for zeljkof so :] [08:01:37] Tobi_WMDE_SWE: separate job template is fine for now, I think [08:02:10] we can merge the job templates later, it is important now that all the jobs are in one place to reduce some code duplication [08:09:28] 3Wikimedia / 3Quality Assurance: mediawiki_selenium always use the same default xvfb display 99 - 10https://bugzilla.wikimedia.org/71602#c4 (10Krinkle) (In reply to Antoine "hashar" Musso from comment #0) > Timo proposed a patch to add chromium and a xvfb server on the contint > slaves which propose to maint... [08:15:35] I don't mind having duplicate code in the SAME repo :-D [08:15:57] dupes are easier to spot this way [08:18:03] (03PS1) 10Krinkle: Don't run oojs-ui-phplint for changes without phpfiles [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165173 [08:18:21] (03PS2) 10Krinkle: Don't run oojs-ui-phplint for changes without php files [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165173 [08:19:29] hashar: there are jobs waiting for over 4 hours already? https://integration.wikimedia.org/ci/ [08:19:54] Tobi_WMDE_SWE: yeah they are throttled at one job per instance [08:19:57] (03CR) 10Krinkle: [C: 032] Don't run oojs-ui-phplint for changes without php files [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165173 (owner: 10Krinkle) [08:20:00] 3Wikimedia / 3Continuous integration: Jenkins: integration-zuul-layoutdiff job says "No layout changes" when there are - 10https://bugzilla.wikimedia.org/71740 (10Krinkle) 3NEW p:3Unprio s:3normal a:3None https://gerrit.wikimedia.org/r/#/c/165173/ reports > integration-zuul-layoutdiff > No layout ch... [08:20:01] so that takes a bunch of hours to process them all [08:20:03] hashar: ah, ok :) [08:20:58] (03Merged) 10jenkins-bot: Don't run oojs-ui-phplint for changes without php files [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165173 (owner: 10Krinkle) [08:21:11] Tobi_WMDE_SWE: we had to throttle them because jobs kill each other X virtual frame buffer that is used to run browser into [08:21:21] + some network connectivity issue with saucelabs [08:21:28] related bug is https://bugzilla.wikimedia.org/show_bug.cgi?id=71602 (xvfb) [08:21:52] zeljkof: maybe we can make the -sauce browser tests to run in parallel again [08:22:26] hashar: it is on my list to test how it works :) [08:22:36] on a few test jobs [08:22:46] (03PS1) 10Krinkle: Run {name}-phplint on labs slaves instead of production slaves [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165175 [08:23:50] was just about to ask why it affects the sauce tests [08:23:58] :) [08:24:56] I think we originally mostly used firefox local [08:25:07] or I just blindly made all browser tests to be throttled :D [08:25:36] hashar: I would assume all jobs ending with "-sauce" to run on saucelabs [08:27:14] (03CR) 10Hashar: [C: 031] "That will free some slots on the production slaves :-]" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165175 (owner: 10Krinkle) [08:27:42] Tobi_WMDE_SWE: that is the case [08:27:50] :) [08:27:58] (03CR) 10Krinkle: [C: 031] "Manually tried with local config change on Jenkins and rebuilding https://integration.wikimedia.org/ci/job/oojs-ui-phplint/128/. Passes fi" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165175 (owner: 10Krinkle) [08:28:14] Krinkle: good morning :] [08:29:00] Krinkle: I could really use your approval on the VisualEditor qunit job made to process submodule https://gerrit.wikimedia.org/r/#/c/163837/12/mediawiki-extensions.yaml,unified :D [08:32:55] (03CR) 10Krinkle: [C: 032] "Deploying:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165175 (owner: 10Krinkle) [08:33:43] hashar: What is "git submodule status" for ? [08:33:56] Krinkle: to show user the sha1 being used for debugging purposes [08:34:11] i though it could avoid some headaches and shows up git submodule update did the job properly [08:34:35] hashar: is this draft currently deployed? [08:34:43] Krinkle: I think so [08:35:16] Krinkle: last run https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/11706/consoleFull [08:35:19] grep for submodule [08:35:25] yeah, there already [08:35:46] if it seems fine to you +1 it , I will rebase the chain of patches [08:36:02] (03Merged) 10jenkins-bot: Run {name}-phplint on labs slaves instead of production slaves [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165175 (owner: 10Krinkle) [08:36:25] hashar: can you maybe poke ops as needed to fix ffs? [08:36:30] phplint is really confusing without it [08:36:31] https://integration.wikimedia.org/ci/job/oojs-ui-phplint/106/console [08:36:36] to find the actual error [08:37:15] I'm "" close to hacking it on labs using a puppetmaster patch that ensures content of /etc/php5/cli/conf.d/fss.ini [08:37:28] Krinkle: from the RT ticket, the patch I wrote is blocked because the package received over updates related to HHVM and nobody is taking the lead to assert that the other changes are fine for Zend PHP. [08:37:49] can we apply just our patch to the version as it is? [08:37:51] Krinkle: I don't really want to spend a couple days to push that forward :/ [08:38:02] theorically we could yeah [08:38:17] but nobody really care about it beside us two and the devs being confused by the warning messages [08:38:32] sounds to me like everybody cares except ops [08:38:34] and I am fed up fighting hours (if not days) to get trivial changes to land :-( [08:39:13] the minutes/hours spent by users looking at the log outweights the 2 minutes it takes for an ops to friggin do it. [08:39:33] if it's not done by tomorrow I'm patching puppetmaster, takes 4 minutes. Which is worth it according to xkcd's scale. [08:40:35] hashar: I'm concerned about the patch as it is (ve qunit) [08:40:37] 07:43:57 Submodule 'lib/ve' () registered for path 'lib/ve' [08:40:48] that means it didn't change anything because it was already correct [08:40:58] which means it has a dirty submodule from the previous job run [08:41:45] Krinkle: you can raise the fss.ini issue on the ops list to get someone to be assigned to it [08:41:52] and since zuul merger doesn't do anything with submodules, that means it is potentially much dirtier than just the wrong sha1. After checking out the new hash (even if forced) there will be untracked files, which can change behaviour of the program (e.g. file_exist, glob() etc.) [08:42:13] we should git-clean it, or re-clone, just like we do with the regular workspace. [08:42:16] ah yeah that is true [08:42:25] probably need to git submodule foreach git reset --hard don't we ? [08:42:38] git reset isn't needed since we do a force check out [08:42:54] but we need clean after the check out to avoid any untracked files [08:43:09] e.g. files present in one hash and untracked, but not present in another hash [08:43:30] (this happens all the time locally, e.g. you create a commit that adds X.txt, then you check out master and you have untracked X.txt) [08:43:37] (03PS1) 10Tobias Gritschacher: Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 [08:44:51] (03PS6) 10Tobias Gritschacher: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 [08:45:06] Krinkle: fss.ini can indeed be fixed via puppet, make sure to reference https://rt.wikimedia.org/Ticket/Display.html?id=7213 and report there a workaround has been made [08:46:38] (03PS7) 10Tobias Gritschacher: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 [08:48:58] hashar: mailed ops :) [08:49:27] hashar: btw [08:49:28] for i in $(ls output/*-phplint); do jenkins-jobs --conf etc/jenkins_jobs.ini update config/ "$(basename $i)"; done; [08:49:50] I imagine you have something similar already, but I hadn't done a 12+ job update before so decided to script it [08:50:46] hehe [08:50:52] you can pass a wildcard to jjb [08:51:01] jenkins-jobs update config/ '*phplint' [08:51:17] this way it only does the yaml -> xml transformation once [08:51:28] thought it would still process the actual update serially [08:51:44] when doing mass changes, I split the list of changes jobs using split -l 100 [08:51:51] that creates batches of 100 which I xargs [08:51:54] or [08:52:13] I just change the XML files manually directly on gallium and reload jenkins. But that is evil hehe [08:52:40] there is a patch pending upstream to have JJB to use threads so it can update jobs in parallel for us [08:55:56] (03PS13) 10Hashar: VisualEditor qunit job now process submodules [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [08:56:48] (03CR) 10Hashar: "Added: git submodule foreach git clean -xqdf" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [09:00:21] !sal [09:00:21] https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:00:35] Yippee, build fixed! [09:00:36] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #228: FIXED in 1 hr 9 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/228/ [09:12:16] hashar: Is there a reason a new precise instance wouldn't provision properly (initial provision) [09:12:32] integration-slave1009 is up and running already but 1004 is still denying public key [09:12:58] maybe because 1004 existed previously? [09:13:08] and when it got deleted some stuff were left behind [09:13:21] integration-slave1004 login: Oct 7 08:58:14 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Name or service not known [09:13:21] Oct 7 08:59:06 integration-slave1004 nslcd[1059]: [16e9e8] ldap_result() failed: No such object [09:13:21] Oct 7 08:59:06 integration-slave1004 nslcd[1059]: [90cde7] ldap_result() failed: No such object [09:13:23] Oct 7 09:00:17 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Name or service not known [09:13:25] Oct 7 09:02:19 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Name or service not known [09:13:27] Oct 7 09:04:23 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Name or service not known [09:13:28] doh [09:13:29] Oct 7 09:05:01 integration-slave1004 nslcd[1059]: [0e0f76] ldap_result() failed: No such object [09:13:31] Oct 7 09:05:01 integration-slave1004 nslcd[1059]: [0e0f76] ldap_result() failed: No such object [09:13:33] Oct 7 09:05:42 integration-slave1004 nslcd[1059]: [9cf92e] ldap_result() failed: No such object [09:13:35] Oct 7 09:06:30 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Temporary failure in name resolution [09:13:37] Oct 7 09:08:32 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Temporary failure in name resolution [09:13:39] Oct 7 09:10:37 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Temporary failure in name resolution [09:13:41] Oct 7 09:11:08 integration-slave1004 nslcd[1059]: [ed7263] ldap_result() failed: No such object [09:13:43] Oct 7 09:12:38 integration-slave1004 puppet-agent[932]: Could not request certificate: getaddrinfo: Temporary failure in name resolution [09:14:03] seems the puppetmaster address is wrong / does not resolve [09:14:07] or might be garbabe [09:14:35] Oct 7 08:55:28 integration-slave1004 nslcd[1059]: [3c9869] ldap_start_tls_s() failed: Can't contact LDAP server: Connection timed out (uri="ldap://virt0.wikimedia.org:389") [09:14:45] Well, there's yar problem [09:14:48] https://wikitech.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=integration&instanceid=b65a604d-40ef-4b16-b527-bfb862ca3904®ion=eqiad [09:15:03] This is a brand new instance [09:15:08] with precise image [09:15:45] AHH [09:15:58] the Precise image probably need to be rebuild / refreshed [09:16:10] I think the default is to use Trusty now [09:16:24] the root cause is: [09:16:24] Can't contact LDAP server: Connection timed out (uri="ldap://virt0.wikimedia.org:389") [09:16:32] Yep [09:16:33] virt0 is no more, it has been replaced by virt1000 [09:16:43] so the Precise image need to be rebuild to reflect that changes [09:16:46] I know, I fixed Jenkins to use it when it locked itself out from access too [09:16:53] that is a bug for labs ops Wikimedia Labs -> Infrastructure [09:17:02] make sure Coren and Andrew B. are cced (they should by default) [09:17:21] and since puppet can't run, it is not going to fix itself :-/ [09:17:30] 1004 instance is moot thus [09:18:35] I know, but I'm working now. I don't coop well with having to file a bug and waiting unknown ETA in order to do my job. If it's a simple fix that shoudl've been fixed as part of maintenance and affecting users in general, and I can't fix it myself and the maintainers are not online, it doesn't exist to me. [09:19:14] hashar: it uses both ldap servers [09:19:17] they fall back [09:19:22] this isn't the root cause [09:19:22] Oct 7 08:55:29 integration-slave1004 nslcd[1059]: [3c9869] connected to LDAP server ldap://virt1000.wikimedia.org:389 [09:20:43] hashar: if I use phplint on Trusty it will use php54 (not php53, and not hhvm) which is kinda inappropiate. [09:23:28] Krinkle: ahh [09:23:36] Krinkle: awesome nice catch :-] [09:23:52] Krinkle: might be worth a comment next to the job template so we remember about it later on [09:24:34] (( and I can't fix it myself and the maintainers are not online, it doesn't exist to me. )) <-- seriously ? [09:24:42] it is never going to be fixed if relevant folks are not made aware of it [09:24:55] and with the Timezone differences you must rely on async tools such as bug filling [09:25:00] or wait for them to show up [09:26:50] hashar: Yes, but it's just that over the past year, *every* *single* *time* i've done *anything* with labs, I end up stopping half-way and file a bug. [09:27:07] Really pisses me off. [09:27:21] Of course I file a bug [09:27:32] I'm just saying for my task right now, it doesn't exist to me. I can't do it right now. [09:27:42] So my 12-level deep stack of what I was doing is wasted. [09:28:39] !log upgrading php5-fss on both beta-cluster and integration instances. {{bug|66092}} https://rt.wikimedia.org/Ticket/Display.html?id=7213 [09:28:43] Logged the message, Master [09:32:14] !log Apply I44d33af1ce85 instead of Ib95c292190d on integration-puppetmaster (remove php5-parsekit package) [09:32:17] Logged the message, Master [09:34:40] !log Rebase integration-puppetmaster on latest operations-puppet (patches: I7163fd38bcd082a1, If2e96bfa9a1c46) [09:34:41] Logged the message, Master [09:51:02] Krinkle: I understand your angriness [09:53:19] !log apt-get upgrade on all beta cluster instances [09:53:20] Logged the message, Master [09:57:27] !log beta: deleting old occurrences of /etc/apt/preferences.d/puppet_base_2.7 [09:57:29] Logged the message, Master [10:00:48] hashar: http://codepen.io/Krinkle/full/mugfw/ [10:10:09] Added diskspace and separated by host (refresh) [10:11:32] Looks like integration-slave1003 is having issues with keeping memory, it's rapidly increasing [10:11:55] 1002 as well. the others are fine [10:12:25] Krinkle: neat [10:12:25] YuviPanda: http://codepen.io/Krinkle/full/mugfw/ :D [10:12:25] YuviPanda: nice monitoring based on graphite metrics [10:12:39] how hard would it be to add it under integration.wikimedia.org somewhere ? [10:12:59] The problem is, this is 20 lines of javascript [10:13:02] that is reallynice [10:13:09] That's all it is. [10:13:40] (see Edit this pad) [10:13:45] yeah seen that [10:14:02] I know YuviPanda plans to add some kind of dashboard to shows the graphite metrics [10:14:11] can't remember the name of the utility that is going to be used though [10:14:17] There is https://github.com/wikimedia/operations-puppet/blob/production/modules/contint/manifests/monitoring.pp [10:14:58] but those are not very useful (e.g. they measure user and system cpu separately, this shows how unsuited graphite is for this purpose, e.g. ganglia and icinga know to add up the different values (except idle)) [10:15:32] and that dashboard isn't happening it seems. I need to know what is going on day-to-day to know stuff isn't going crazy (but not critical yet) [10:16:00] I just want ganglia back, or something that comes really close in terms of just seeing how memory/cpu/disk/network is doing at the moment overall and per-node. [10:16:44] ah http://grafana.wikimedia.org/#/dashboard/db/edit-performance [10:16:44] meant to replace gdash [10:16:46] !log beta: apt-get upgraded all instances beside the lucid one. [10:16:48] Logged the message, Master [10:17:01] ganglia will be phased out [10:17:22] and nobody is going to resurrect it on labs. I think the related labs project has been deleted as well [10:18:03] hashar: I'm confident that production ganglia is not going anyway unless and until there is a usable and reliable replacement. [10:18:11] the monitoring presentation layer is being handled by YuviPanda , you can poke it to get more infos. He pasted me a roadmap at some point but I lost /forgot it [10:18:31] So far all I've seen is graphite queries that aren't very useful or maintainable. But we'll see. Maybe it's like regex, just takes someone to really dig in and write it right, and then magic happens. [10:20:43] I am out for lunch with kid(2) and wife() :-D [10:21:41] Krinkle: yeah you can ask graphite for a sum of some wildcard tree [10:21:52] something like labs.integration.integration-slave*.cpu.count [10:21:56] or something along those lines [10:22:02] lunchiiing [10:22:07] hashar: I know, that's what I use to display a single metric for multiple sources [10:22:19] but what I need is to combine indivual metrics for a single source [10:22:24] e.g. 100-cpu.idle [10:22:33] or sum(cpu.user, cpu.system) [10:22:56] or whatever it is that Ganglia and other normal monitoring tools use when they refer to "CPU usage" [10:23:05] This is basic stuff. [10:28:49] (03CR) 10Zfilipin: [C: 04-1] Make repository host parameterizable (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [10:29:28] Tobi_WMDE_SWE: let me know if you have questions about ^ [10:34:28] (03CR) 10Zfilipin: [C: 032] Additional headless environment variables [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) (owner: 10Dduvall) [10:34:45] (03Merged) 10jenkins-bot: Additional headless environment variables [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) (owner: 10Dduvall) [10:39:19] (03CR) 10Zfilipin: Create mediawiki-selenium-bundle-rspec (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165083 (owner: 10Hashar) [10:42:42] Krinkle: so, you could either just set it to monitor that idle isn't under a certain percentage... [10:42:47] Krinkle: or sum it up using functions from http://graphite.readthedocs.org/en/latest/functions.html [10:44:30] YuviPanda: I've looked for a way to do math (100-idle) or addition (a+b) but haven't found any [10:44:38] I found sum() but it seems to only take a single pattern [10:44:47] not two targets or two patterns [10:45:01] Krinkle: are you looking at this for monitoring or for graphs? [10:45:08] monitoring [10:45:17] But monitoring is useless without graphs [10:45:38] true [10:45:51] I don't need an e-mail saying CPU went 99%, I want to see it grow over two days time, and if nobody noticed, get an e-mail, look at the graph and know how it grew [10:46:17] current monitoring only gives me boolean values in icinga [10:47:05] Krinkle: I do not think we use sikuli at all any more https://gerrit.wikimedia.org/r/#/c/163791/18/modules/contint/manifests/browsertests.pp,unified [11:11:38] (03CR) 10Tobias Gritschacher: "@Zfilipin:" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [11:19:13] (03CR) 10Hashar: "Lovely https://doc.wikimedia.org/rubygems/mediawiki-selenium/#Headless_Mode" [selenium] - 10https://gerrit.wikimedia.org/r/164704 (https://bugzilla.wikimedia.org/71602) (owner: 10Dduvall) [11:21:50] (03PS1) 10Krinkle: integration: Remove broken images [integration/docroot] - 10https://gerrit.wikimedia.org/r/165186 [11:21:52] (03PS1) 10Krinkle: integration: Remove dashboard and integrate into Jenkins views [integration/docroot] - 10https://gerrit.wikimedia.org/r/165187 [11:21:54] (03PS1) 10Krinkle: Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 [11:22:04] (03CR) 10jenkins-bot: [V: 04-1] Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 (owner: 10Krinkle) [11:22:19] (03PS1) 10Krinkle: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 [11:22:21] (03CR) 10jenkins-bot: [V: 04-1] integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 (owner: 10Krinkle) [11:22:28] (03PS2) 10Krinkle: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 [11:22:30] (03CR) 10jenkins-bot: [V: 04-1] integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 (owner: 10Krinkle) [11:23:18] (03PS2) 10Krinkle: Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 [11:23:20] (03PS3) 10Krinkle: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 [11:23:22] (03CR) 10Hashar: [C: 032] "Indeed no point in keeping them on the main page. https://integration.wikimedia.org/dashboard/ is meant for that and has been updated prop" [integration/docroot] - 10https://gerrit.wikimedia.org/r/165186 (owner: 10Krinkle) [11:23:24] (03Merged) 10jenkins-bot: integration: Remove broken images [integration/docroot] - 10https://gerrit.wikimedia.org/r/165186 (owner: 10Krinkle) [11:23:26] (03CR) 10jenkins-bot: [V: 04-1] Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 (owner: 10Krinkle) [11:23:28] (03CR) 10jenkins-bot: [V: 04-1] integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 (owner: 10Krinkle) [11:23:58] (03PS3) 10Krinkle: Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 [11:24:00] (03PS4) 10Krinkle: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 [11:24:03] (03CR) 10jenkins-bot: [V: 04-1] integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 (owner: 10Krinkle) [11:24:48] (03PS5) 10Krinkle: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 [11:25:29] Krinkle: the VE-qunit job now has git submodule foreach git clean -xqdf [11:25:50] i guess that address potentially remaining files when submodule is being changed ( https://gerrit.wikimedia.org/r/#/c/163837/13/mediawiki-extensions.yaml,unified ) [11:26:11] before or after submodule update checkout? [11:27:13] after [11:27:28] you can git submodule foreach if no module has been registered yet [11:28:51] (03CR) 10Hashar: [C: 04-1] "There is stil a link to dashboard/ floating around (in the H4 header on the main page). Otherwise fine to me (less duplication)." (031 comment) [integration/docroot] - 10https://gerrit.wikimedia.org/r/165187 (owner: 10Krinkle) [11:29:41] post lunch gaming session brb [11:31:32] Project browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce build #1: FAILURE in 12 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/1/ [11:32:57] hashar: after you are back from gaming, can you help me understand? [11:33:10] in https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/1/console [11:33:10] Tobi_WMDE_SWE: sure in roughly 20 mins [11:33:22] hashar: ok! [11:33:26] hmm paused game [11:33:43] (03PS6) 10Krinkle: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 [11:33:50] Tobi_WMDE_SWE: sounds like a pass isue [11:33:53] I'm doing cp /mnt/jenkins-workspace/workspace/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/tests/browser/config/config_ci.yml /mnt/jenkins-workspace/workspace/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/tests/browser/config/config.yml [11:34:03] to copy the config.yml [11:34:12] ls /mnt/jenkins-workspace/workspace/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/tests/browser/config/ [11:34:14] shows it is there [11:34:43] (03PS4) 10Krinkle: Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 [11:34:45] (03PS7) 10Krinkle: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 [11:34:46] but when running the tests it is gone.. also when I look into the workspace, it is not there [11:34:47] (03PS2) 10Krinkle: integration: Remove dashboard and integrate into Jenkins views [integration/docroot] - 10https://gerrit.wikimedia.org/r/165187 [11:34:58] (03CR) 10Krinkle: "Fixed." [integration/docroot] - 10https://gerrit.wikimedia.org/r/165187 (owner: 10Krinkle) [11:35:03] and we did cd to tests/browser [11:35:11] hashar: yes [11:35:19] it is not in the workspace [11:35:27] (03CR) 10Hashar: [C: 032] ":-]" [integration/docroot] - 10https://gerrit.wikimedia.org/r/165187 (owner: 10Krinkle) [11:35:29] it get's cleaned or something [11:35:29] (03Merged) 10jenkins-bot: integration: Remove dashboard and integrate into Jenkins views [integration/docroot] - 10https://gerrit.wikimedia.org/r/165187 (owner: 10Krinkle) [11:35:46] Krinkle: you can land the other changes as well and iterate as needed if anything is wrong :] [11:35:52] sure [11:36:02] (03CR) 10Krinkle: [C: 032] Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 (owner: 10Krinkle) [11:36:04] (03Merged) 10jenkins-bot: Move shared/bootstrap to shared/lib/bootstrap [integration/docroot] - 10https://gerrit.wikimedia.org/r/165188 (owner: 10Krinkle) [11:36:15] hashar: Since I fixed the path inclusions, you can test these locally btw [11:36:25] just make sure the repo is somewhere in your localhost docroot and it works [11:36:36] e.g. localhost/wikimedia/integration/docroot/org/wikimedia/integration/ [11:36:47] Tobi_WMDE_SWE: in the shell you can add a find command such as : find $WORKSPACE -name config.yml [11:36:52] That's how I develop it :) [11:36:54] (03PS2) 10Tobias Gritschacher: Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 [11:37:01] Tobi_WMDE_SWE: that will help diagnostic and figure out where the file ends up being generated [11:37:03] (03CR) 10Krinkle: [C: 032] integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 (owner: 10Krinkle) [11:37:05] (03Merged) 10jenkins-bot: integration: Implement Monitoring page [integration/docroot] - 10https://gerrit.wikimedia.org/r/165189 (owner: 10Krinkle) [11:37:13] Tobi_WMDE_SWE: oh found it [11:37:24] Tobi_WMDE_SWE: the file is copied before the workspace is being cleaned up [11:37:33] hashar: it is inside the config/ directory after I copied it.. [11:37:51] but it is gone when the tests are executed [11:37:57] Tobi_WMDE_SWE: after theJenkins git plugin has been run , at one point we query the target host for the git_branch being used there and checkout -f thatbranch [11:38:13] Tobi_WMDE_SWE: that is to make the job branch to match the target wiki [11:38:15] (03PS3) 10Tobias Gritschacher: Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 [11:38:28] https://integration.wikimedia.org/monitoring/ :) [11:38:39] Krinkle: awesome! [11:38:52] Krinkle: that is worth an announce on the QA list I believe [11:38:54] I added an overview with sum() on top (compared to the codepen) [11:39:12] so it'll have like 700 CPU, but still useful I think [11:39:14] yeah that is great [11:40:22] hashar: uhm.. [11:40:31] Tobi_WMDE_SWE: yeah sorry :( [11:40:39] that happens in the browsertest-website builder [11:40:47] Tobi_WMDE_SWE: you can move that config.yml copying to the browsertests-website [11:41:03] Tobi_WMDE_SWE: i.e. if tests/browser/config/config_ci.yml exists, do the copy :] [11:41:18] or maybe it can just be renamed ? [11:41:22] hashar: that is very wikidata specific.. so I don't think it should go there [11:42:04] or is there need in other projects for that? [11:42:28] what i mean is that if cucumber look for a config/config.yml [11:42:36] you might probably rename the file in your repository [11:43:00] hashar: it works more like the LocalSettings.php in wikis [11:43:01] we used to have a config.yml in most repositories to let dev set tags and over things [11:43:03] but it has been removed [11:43:15] there is a config.yml.sample [11:43:45] every user of the tests can have his own config.yml [11:43:50] hashar: sent [11:43:54] Tobi_WMDE_SWE: yeah that is true [11:43:57] Krinkle: awesome :] [11:44:24] (03CR) 10Zfilipin: "Is there a reason this is not yet merged into master?" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 (owner: 10Tobias Gritschacher) [11:44:34] Tobi_WMDE_SWE: maybe we can establish the convention that whenever a repo lack a config.yml we would copy the config_ci.yml [11:45:00] Tobi_WMDE_SWE: talk with zeljkof about providing custom cucumber profile file. he probably has a better idea [11:45:16] hashar: hm, yes. will do. thanks! [11:45:26] will be back in ~15 minutes [11:45:35] gotta finish that level *grin* [11:45:45] hashar: have fun! ;) [11:45:48] hasharGame: what are you playing? :) [11:45:51] zeljkof: I've commented on https://gerrit.wikimedia.org/r/#/c/164332/ [11:46:13] Tobi_WMDE_SWE: saw that, will test locally [11:46:18] ok [11:46:31] did not know that it has to be defined on project level [11:46:42] that new jjb feature is confusing [11:46:43] zeljkof: hashar found that out [11:46:56] I think he implemented the upstream feature :) [11:46:56] I was messing around hours.. [11:47:06] Ugh, puppet failing for browsertests class [11:47:27] https://gist.githubusercontent.com/Krinkle/298016a6fef76d690d9a/raw/ [11:49:28] Notice: /Stage[main]/Labs_lvm/Exec[create-volume-group]/returns: Error: Can't create any more partitions. [11:49:29] ? [11:50:31] Hm.. /mnt/ exists but empty, except for /mnt/redis/integration-slave1009-6379 [11:51:48] (03CR) 10Krinkle: [C: 04-1] Move wikidata-performance browsertests job to WMF Jenkins (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 (owner: 10Tobias Gritschacher) [11:52:32] zeljkof: some random basic game http://www.kongregate.com/games/whiteboardwar/creeper-world-3-abraxis [11:52:37] I am done already (lost) [11:53:04] !log Deleted integration-slave1004 because {{bug|71741}} [11:53:06] Logged the message, Master [11:53:15] zeljkof: the user variable defined in a defaults section can't be used in that default :( [11:53:49] hashar: yeah, I remember misunderstanding that before :) [11:54:07] !log The new integration-slave1009 must remain unpooled because Setup failed (puppet unable to mount /mnt) - see also [[Nova Resource:Integration/Setup]] [11:54:09] Logged the message, Master [11:54:22] zeljkof: but if we apply it at the project level, the variable defines in the defaults are known by the project and passed to the default [11:55:12] (03PS12) 10Hashar: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 [11:55:25] * zeljkof will be right back [11:55:43] (03PS14) 10Hashar: VisualEditor qunit job now process submodules [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 [11:56:29] (03PS2) 10Hashar: Mark mw-setup-extension as obsolete [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161460 [11:56:38] (03CR) 10Hashar: "rebased" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161460 (owner: 10Hashar) [11:57:21] Krinkle: re VE - qunit change , can i get it merged now ? :D https://gerrit.wikimedia.org/r/#/c/163837/14/mediawiki-extensions.yaml,unified [12:03:15] 3Wikimedia / 3Continuous integration: Fix "PHP Deprecated: Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/fss.ini" on integration slaves - 10https://bugzilla.wikimedia.org/66092#c2 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX The new php5-fss package has been build and uploaded on apt.wi... [12:06:14] 3Wikimedia / 3Continuous integration: Add Jenkins jobs for iegreview application - 10https://bugzilla.wikimedia.org/71617#c6 (10Antoine "hashar" Musso) With bug 71642 being fixed (JenkinsBot has been given merge right on the whole wikimedia/* tree). I guess Jenkins jobs now works fine for iegreview :] [12:09:57] (03CR) 10Hashar: [C: 032] "All jobs updated (beside VisualEditor one)" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 (owner: 10Hashar) [12:10:18] (03CR) 10Hashar: [C: 032] "Deployed, seems to work fine. Can amend later on if needed." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [12:13:14] (03Merged) 10jenkins-bot: Switch extensions qunit jobs to Zuul cloner [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161459 (owner: 10Hashar) [12:13:49] (03Merged) 10jenkins-bot: VisualEditor qunit job now process submodules [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [12:13:51] (03Merged) 10jenkins-bot: Mark mw-setup-extension as obsolete [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/161460 (owner: 10Hashar) [12:18:40] (03PS1) 10Hashar: Revert "Don't run oojs-ui-phplint for changes without php files" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165191 [12:18:51] (03CR) 10Hashar: [C: 032] Revert "Don't run oojs-ui-phplint for changes without php files" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165191 (owner: 10Hashar) [12:18:59] (03Merged) 10jenkins-bot: Revert "Don't run oojs-ui-phplint for changes without php files" [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165191 (owner: 10Hashar) [12:33:29] (03PS4) 10Hashar: Remove legacy mediawiki core jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163820 [12:37:12] (03CR) 10Hashar: [C: 032] "Jobs deleted \O/" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163820 (owner: 10Hashar) [12:38:43] 3Wikimedia / 3Continuous integration: [upstream] Jenkins: deadlock when deleting jobs - 10https://bugzilla.wikimedia.org/51802#c7 (10Antoine "hashar" Musso) 5NEW>3RESO/FIX I had the opportunity to delete several jobs each having a long history. No deadlock appeared with Jenkins 1.565.3 [12:39:34] (03PS2) 10Hashar: browsertests: enable Performance publisher [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 [12:40:25] (03Merged) 10jenkins-bot: Remove legacy mediawiki core jobs [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163820 (owner: 10Hashar) [12:41:56] zeljkof: I am going to enable the PerformancePlugin Tobi_WMDE_SWE suggested :-D ( change is https://gerrit.wikimedia.org/r/#/c/163857/ ) [12:45:44] (03PS3) 10Hashar: browsertests: enable Performance publisher [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 [12:45:59] (03CR) 10Hashar: [C: 032] "Reconfiguring all jobs." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 (owner: 10Hashar) [12:46:59] (03CR) 10jenkins-bot: [V: 04-1] browsertests: enable Performance publisher [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 (owner: 10Hashar) [12:47:01] hashar: go ahead [12:47:05] let's see what happens :) [12:47:37] zeljkof: jjb is still slowly updating them all [12:47:51] browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce has been updated :-D [12:48:00] I guess we will see the result with this evening run [12:48:26] hashar: it probably needs two test runs to show some data [12:48:35] probably [12:49:12] (03Merged) 10jenkins-bot: browsertests: enable Performance publisher [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163857 (owner: 10Hashar) [12:52:45] (03CR) 10Zfilipin: Create mediawiki-selenium-bundle-rspec (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165083 (owner: 10Hashar) [12:53:37] (03CR) 10Zfilipin: [C: 031] Create mediawiki-selenium-bundle-rspec [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165083 (owner: 10Hashar) [13:05:45] (03PS5) 10Tobias Gritschacher: Move wikidata-performance browsertests job to WMF Jenkins [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 [13:07:49] (03PS1) 10Tobias Gritschacher: Fix node for Wikidata browsertest job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165195 [13:08:00] (03CR) 10Tobias Gritschacher: Move wikidata-performance browsertests job to WMF Jenkins (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163129 (owner: 10Tobias Gritschacher) [13:11:33] zeljkof: can you revisit https://gerrit.wikimedia.org/r/#/c/164332/? [13:12:32] Tobi_WMDE_SWE: sure, finishing the last review in a few minutes, will take a look then [13:12:50] I mean, I am reviewing something else now :) [13:12:57] will take a look in a few minutes [13:13:27] zeljkof: thx [13:15:09] Tobi_WMDE_SWE: ok, the last -1 for today, I hope :) [13:15:15] let's see your changes [13:26:56] performance plugin enabled everywhere :D [13:28:38] (03PS2) 10Hashar: Zuul cloner now defaults --branch to $ZUUL_BRANCH [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164958 [13:30:45] (03CR) 10Zfilipin: [C: 032] Fix node for Wikidata browsertest job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165195 (owner: 10Tobias Gritschacher) [13:32:53] (03CR) 10Zfilipin: [C: 032] Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [13:33:07] Tobi_WMDE_SWE: ^ [13:33:18] argh [13:33:23] merge conflict [13:33:28] let me try rebasing [13:33:33] (03PS8) 10Zfilipin: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [13:33:49] (03Merged) 10jenkins-bot: Fix node for Wikidata browsertest job [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165195 (owner: 10Tobias Gritschacher) [13:33:51] (03CR) 10Zfilipin: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [13:34:06] (03PS9) 10Zfilipin: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [13:34:12] (03CR) 10Zfilipin: [C: 032] Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [13:34:26] ok, that should merge it [13:34:32] zeljkof: \o/ [13:34:36] (03CR) 10Hashar: [C: 032] "Reconfiguring a first pass, will do the testextension ones later" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164958 (owner: 10Hashar) [13:37:25] (03Merged) 10jenkins-bot: Make repository host parameterizable [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164332 (owner: 10Tobias Gritschacher) [13:41:52] zeljkof: do I understand correctly: all scenarios should be tagged with the browsers they should be tested with? [13:42:24] so, if I want my scenarios to be tested with firefox, I have to add @firefox [13:42:32] Tobi_WMDE_SWE: yes, that is how we determine which jobs to run where (browser/site combinations) [13:42:38] lot of changes are landing in apparently [13:42:40] that is nice [13:42:41] can you link me to the documentation? [13:42:54] Tobi_WMDE_SWE: yes, but only to the top of the file [13:43:23] zeljkof: ok, then I understood correctly [13:43:51] Tobi_WMDE_SWE: https://www.mediawiki.org/wiki/Manual:Coding_conventions/Selenium#Required_tags [13:43:52] some docs [13:44:31] (03PS3) 10Hashar: Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 [13:44:39] zeljkof: thx! I was not able to find this.. [13:48:15] (03CR) 10Hashar: [C: 032] Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 (owner: 10Hashar) [13:48:35] damn conflicts [13:48:41] (03PS3) 10Hashar: Zuul cloner now defaults --branch to $ZUUL_BRANCH [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164958 [13:48:53] (03CR) 10Hashar: [C: 032] "Deploying everything" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164958 (owner: 10Hashar) [13:48:59] (03PS4) 10Hashar: Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 [13:49:05] (03CR) 10Hashar: [C: 032] Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 (owner: 10Hashar) [13:49:17] (03CR) 10jenkins-bot: [V: 04-1] Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 (owner: 10Hashar) [13:49:20] (03CR) 10jenkins-bot: [V: 04-1] Zuul cloner now defaults --branch to $ZUUL_BRANCH [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164958 (owner: 10Hashar) [13:50:18] Tobi_WMDE_SWE: yeah, not the easiest thing to find :( [13:50:34] my plan is to work on the docs too, in the near future(tm) [13:51:15] Project browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce build #3: ABORTED in 1 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/3/ [13:52:37] (03Merged) 10jenkins-bot: Zuul cloner now defaults --branch to $ZUUL_BRANCH [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164958 (owner: 10Hashar) [13:53:07] (03Merged) 10jenkins-bot: Test integration/phpunit against mw core branches [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164959 (owner: 10Hashar) [13:57:59] hashar: zeljkof: \o/ wikidata tests are now using the templates [13:58:03] https://integration.wikimedia.org/ci/view/BrowserTests/view/Wikidata/job/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/4/console [13:58:29] Tobi_WMDE_SWE: congratulations! [14:02:10] (03PS4) 10Tobias Gritschacher: Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 [14:06:00] (03PS5) 10Tobias Gritschacher: Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 [14:08:57] (03PS10) 10Hashar: Bundle macro + ruby doc using yard [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160983 [14:10:06] (03PS2) 10Hashar: Create mediawiki-selenium-bundle-rspec [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165083 [14:14:06] (03CR) 10Hashar: "Lets unleash this bundle macro. We can look at migrating the bundle jobs to Trusty/ruby2 in another change." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160983 (owner: 10Hashar) [14:14:14] (03CR) 10Hashar: [C: 032] Bundle macro + ruby doc using yard [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160983 (owner: 10Hashar) [14:14:22] (03PS6) 10Tobias Gritschacher: Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 [14:16:21] hashar: zeljkof: here's the change to make wikidata tests use the templates: https://gerrit.wikimedia.org/r/#/c/165177/ [14:16:56] it would not work for the performance-tests job yet as that would not work out with the current tagging schema [14:17:17] e.g. it is just one specific feature I want to run [14:17:32] but I will think about it tomorrow [14:17:37] (03Merged) 10jenkins-bot: Bundle macro + ruby doc using yard [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/160983 (owner: 10Hashar) [14:19:19] (03CR) 10Hashar: "rebasing" (032 comments) [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [14:19:22] (03CR) 10Zfilipin: [C: 031] Make Wikidata jobs use the templates (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 (owner: 10Tobias Gritschacher) [14:19:25] (03PS2) 10Hashar: WIP Environment abstraction layer [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [14:19:38] (03CR) 10Hashar: "check experimental" [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [14:19:43] (03CR) 10jenkins-bot: [V: 04-1] WIP Environment abstraction layer [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [14:21:23] (03CR) 10Hashar: "Oh men:" [selenium] - 10https://gerrit.wikimedia.org/r/159644 (owner: 10Dduvall) [14:25:17] (03PS7) 10Tobias Gritschacher: Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 [14:25:44] (03CR) 10Tobias Gritschacher: Make Wikidata jobs use the templates (031 comment) [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 (owner: 10Tobias Gritschacher) [14:25:51] zeljkof: Tobi_WMDE_SWE I opened a new threads on the QA list to get the integration/*-config repos merged together. feedback welcome! [14:26:12] hashar: saw that, will reply with +1 :) [14:28:59] (03PS3) 10Hashar: Jobs for search/extra [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164316 [14:31:44] (03CR) 10Zfilipin: [C: 031] Make Wikidata jobs use the templates [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165177 (owner: 10Tobias Gritschacher) [14:35:43] 3Wikimedia / 3Quality Assurance: Add Sikuli to the machines that run browser tests - 10https://bugzilla.wikimedia.org/54393 (10Ċ½eljko Filipin) 5PATC>3RESO/WON [14:48:59] (03PS1) 10Tobias Gritschacher: Allow setting the browser-timeout [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165209 [14:49:34] manybubbles: search/extra doc will self update at https://doc.wikimedia.org/search-extra/ soon :-D [14:49:48] hashar: you are amazing! [14:50:04] (03PS4) 10Hashar: Jobs for search/extra [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164316 [14:50:32] That makes me want to write more javadoc which is never a bad thing [14:50:43] (03CR) 10Hashar: [C: 032] "Job will publish doc at https://doc.wikimedia.org/search-extra/ . Will add related zuul trigger." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164316 (owner: 10Hashar) [14:51:44] (03PS5) 10Hashar: Jobs for search/extra [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164316 (https://bugzilla.wikimedia.org/71662) [14:51:59] (03PS2) 10Tobias Gritschacher: Allow setting the browser-timeout in browsertests [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165209 [14:52:05] (03CR) 10Hashar: [C: 032] Jobs for search/extra [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164316 (https://bugzilla.wikimedia.org/71662) (owner: 10Hashar) [14:52:38] (03PS1) 10Hashar: Trigger search-extra-javadoc-publish [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165211 (https://bugzilla.wikimedia.org/71662) [14:53:26] manybubbles: ho I would be amazing if i managed to lead all the devs to write the jenkins jobs themselves :D [14:53:52] hashar: I almost was doing it then I got distracted with other things [14:54:03] (03CR) 10Hashar: [C: 032] Trigger search-extra-javadoc-publish [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165211 (https://bugzilla.wikimedia.org/71662) (owner: 10Hashar) [14:54:12] (03Merged) 10jenkins-bot: Trigger search-extra-javadoc-publish [integration/zuul-config] - 10https://gerrit.wikimedia.org/r/165211 (https://bugzilla.wikimedia.org/71662) (owner: 10Hashar) [14:55:27] (03Merged) 10jenkins-bot: Jobs for search/extra [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/164316 (https://bugzilla.wikimedia.org/71662) (owner: 10Hashar) [14:56:30] 3Wikimedia / 3Continuous integration: Create documentation publish job for search-extra - 10https://bugzilla.wikimedia.org/71662#c5 (10Antoine "hashar" Musso) 5PATC>3RESO/FIX a:3Antoine "hashar" Musso I manually generated documentation from the master branch. The job will be triggered on post merge and... [14:56:45] manybubbles: Yuvi also added some check style target [14:56:56] for some other job I can't remember of [14:57:24] hashar: generally its easier to add that to maven and have it fail the job if the checkstyle fails. You don't get graphs in jenkins but you can easily test it locally [14:57:34] maven is _supposed_ to centralize all the configuration. [14:57:52] yup it is kind of nice [14:58:00] maybe php will have such a horse one day [15:10:21] having a break before the weekly checkin [15:38:13] g'morn [15:41:02] greg-g: that looks weird on my client. I see 'greg-g g'morn' [15:41:08] Too many g's :p [15:42:50] :) [15:42:58] ggggggggg'morning [15:43:40] greg-g: if you added the g to morning on purpose - get running :p [15:45:59] 3Wikimedia Labs / 3deployment-prep (beta): Determine first pass list of icinga-alerting data from graphite.wmflabs - 10https://bugzilla.wikimedia.org/70141#c18 (10Greg Grossmeier) 5ASSI>3RESO/FIX (In reply to Greg Grossmeier from comment #17) > Yuvi: Thanks for the first pass work! Once you remove yourse... [15:54:30] Project browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce build #6: STILL FAILING in 1 hr 3 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-WikidataTests-wikidata.beta.wmflabs.org-linux-firefox-sauce/6/ [18:24:12] !log /var/lib/jenkins-slave/tmpfs full (100%) on gallium [18:24:17] Logged the message, Master [19:07:38] Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #216: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/216/ [19:11:53] Hey QA folks, fyi I got this log: https://integration.wikimedia.org/ci/job/mwext-MultimediaViewer-qunit/2582/console [19:12:01] > 18:15:42 Could not insert main page: A database error has occurred. Did you forget to run maintenance/update.php after upgrading? See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script [19:12:06] Seems bad [19:12:15] But it didn't happen when I resubmitted, so I'm OK for now [19:18:34] marktraceur: The scratch disk on gallium was 100% full. ^d just nuked the tmp db files there to make space for things again [19:19:23] ah we have a bug for that [19:19:30] !log ^d deleted all files/directories in gallium:/var/lib/jenkins-slave/tmpfs [19:19:34] Logged the message, Master [19:19:35] happens whenever we have more than N qunit jobs running over an hour :/ [19:20:03] Yeah. qunit was soaking up most of the tiny .5G ram disk [19:20:26] Reedy opened an RT ticket asking for more ram [19:20:55] Gallium is tiny for the jobs it it trying to do [19:21:08] My laptop has more ram and cpu [19:23:28] tis only 8GB iirc [19:23:32] 3+ years old [19:24:27] but yeah some more RAM would probably help [19:25:01] Aha. [19:26:17] surely having to populate a huge object cache using a sqlite db is not very nice either [19:37:15] 3Wikimedia / 3Continuous integration: Jenkins: lanthanum/gallium tmpfs are filling up with stale tmp files - 10https://bugzilla.wikimedia.org/71128 (10Antoine "hashar" Musso) [19:37:26] Reedy: there is the bug ^^^ :D [19:39:00] 3Wikimedia / 3Continuous integration: Jenkins: lanthanum/gallium tmpfs are filling up with stale tmp files - 10https://bugzilla.wikimedia.org/71128#c4 (10Sam Reed (reedy)) RT 8582 for requesting more ram in those 2 machines [19:44:30] (03PS1) 10Hashar: Remove '-qunit' from 'prepare-mediawiki-qunit' [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165321 [19:58:09] (03PS1) 10Hashar: Avoid dupe code by using prepare-mediawiki macro [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165327 [19:59:53] (03CR) 10Krinkle: "Use -qdff (double f) as the new standard pattern (see also https://github.com/wikimedia/integration-jenkins/commit/08147c42ea42e1a5) to av" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [20:00:32] Yippee, build fixed! [20:00:32] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #106: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/106/ [20:01:00] 3Wikimedia / 3Continuous integration: Jenkins: lanthanum/gallium tmpfs are filling up with stale tmp files - 10https://bugzilla.wikimedia.org/71128#c5 (10Antoine "hashar" Musso) We can use a postbuilder publisher that execute a shell script to teardown the database. Should be done in jobs creating sqlite d... [20:01:47] Krinkle: had to revert your phplint jobs optimization :-/ Reasonning is in commit message of revert. [20:02:00] (03PS1) 10Krinkle: mwext-VisualEditor-qunit: Use git clean "-ff" instead "-f" [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/165328 [20:02:06] (03CR) 10Krinkle: "Fixed in I38fde56213f7e63a3d28859dd369a9bf1c5fae68." [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163837 (owner: 10Hashar) [20:02:09] Krinkle: some jobs are only triggered if phplint job pass and if phplint is not triggered the childs are not and stick in the queue :/ [20:03:50] hashar: OK. How about removing that tree structure? [20:04:13] Krinkle: the idea if that if phplint doesn't pass, there is no point in running the long running tests. [20:04:32] I've done it elsewhere as well. I think it's a dated principle that only optimises a very uncommon use case, and will delay the 99% usecase by having to wait for a slave, wordspace, git clone etc. for phplint and then do it al over again [20:04:40] that can be happening in parellel otherwise [20:04:46] I know, but I don't think that matters. [20:04:53] it'll fail early enough [20:05:42] not really [20:05:54] the tests might well never load the faulty PHP file [20:06:21] and I am not sure what is the gain in not triggering the phplint jobs [20:06:30] even better in that case [20:06:48] this is like being able to get 2 phpunit errors instead of 1 (with the other only triggered after you fix the first) [20:07:15] more productivity if I know phpunit passes (or if it doesn't and has a regular unit test failure, I know which one and can fix it) [20:07:31] not triggering the job is just common practice, we do taht everywhere, you started that. [20:08:04] It's not a hugh deal, but it doesn't have any downsides. And saves a slot in the slaves, creation of workspace, zuul-merger, zuul posting message to Jenkins etc. [20:08:36] we also run phpcs separate from phplint, should that be made dependent? [20:08:36] so you want to save 2 seconds overlay (the typical run time for mediawiki-core-phplint ) but potentially waste several minutes even when we know a file does not pass php -l ? [20:08:39] doesn't make sense to me [20:08:57] yeah phpcs can probably be made dependent as well [20:09:27] hashar: well, you know where it is going eventually: one job that does it all. [20:09:39] then it makes sense to do them in sequence by default, and internally parallel where possible [20:10:07] e.g. npm/grunt does jshint -> jscs ->uglify, less->csslint->concat->cssjanus etc. [20:10:26] but as long as they're separate jobs.. [20:10:55] mediawiki-core-test: builders: phplint, phpcs, .. [20:11:40] anyway, I'll leave it. no time, no priority [20:17:30] Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #231: FAILURE in 1 hr 8 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/231/ [20:24:42] hashar: https://bugzilla.wikimedia.org/show_bug.cgi?id=71128 is imho critical as it brings down gallium/lanthanum every other day. [20:25:11] Please asssign to me or yourself, this should imho be fixed by the end of this week. Just a teardown handler for the macro, right? [20:26:15] hashar: greg-g: Sorry for missing the releng standup, am mixed up after the jetlag. Anything in particular I should know for this/last week? [20:27:38] Krinkle: will assign bug 71128 to me [20:27:42] ok :) [20:28:01] Krinkle: I have a clear idea bout how to fix it ( publisher: post builder: shell: rm :-] ) [20:31:16] 3Wikimedia / 3Continuous integration: Jenkins: lanthanum/gallium tmpfs are filling up with stale tmp files - 10https://bugzilla.wikimedia.org/71128#c6 (10Antoine "hashar" Musso) s:5critic>3normal a:3Antoine "hashar" Musso That is surely annoying but not that critical imho. Now that I have completed t... [20:31:27] Krinkle: your https://integration.wikimedia.org/monitoring/ has been praised :] [20:31:50] I have skip announcing your wonderful java/maven hacking of gitplugin from yesterday [20:32:06] though it is in the memo [20:33:26] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #247: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/247/ [20:36:03] Yippee, build fixed! [20:36:03] Project browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce build #184: FIXED in 1 min 35 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-test2.wikipedia.org-linux-firefox-sauce/184/ [20:36:23] (03CR) 10Reedy: "4.3 is now stable..." [integration/phpunit] - 10https://gerrit.wikimedia.org/r/151252 (owner: 10EBernhardson) [20:37:26] Krinkle: have you deleted the integration-dev instance ? [20:38:13] Yippee, build fixed! [20:38:14] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #92: FIXED in 9 min 9 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/92/ [20:39:55] oh arhaerheaheeeee [20:40:12] dev Jenkins disappeared :-/ [20:40:14] off to bed [20:41:19] (03PS1) 10Krinkle: integration/monitoring: Clean up implementation [integration/docroot] - 10https://gerrit.wikimedia.org/r/165342 [20:43:07] (03PS2) 10Krinkle: integration/monitoring: Clean up implementation [integration/docroot] - 10https://gerrit.wikimedia.org/r/165342 [20:49:30] (03PS3) 10Krinkle: integration/monitoring: Clean up implementation [integration/docroot] - 10https://gerrit.wikimedia.org/r/165342 [20:49:30] (03CR) 10Krinkle: [C: 032] integration/monitoring: Clean up implementation [integration/docroot] - 10https://gerrit.wikimedia.org/r/165342 (owner: 10Krinkle) [20:49:31] (03Merged) 10jenkins-bot: integration/monitoring: Clean up implementation [integration/docroot] - 10https://gerrit.wikimedia.org/r/165342 (owner: 10Krinkle) [21:00:16] 3Wikimedia / 3Continuous integration: CI: integration-puppetmaster < 300MB free memory (ganglia and puppet take up resources) - 10https://bugzilla.wikimedia.org/71756 (10Krinkle) 3NEW p:3Unprio s:3normal a:3None See https://integration.wikimedia.org/monitoring/#h-integration-puppetmastermem These ar... [21:03:33] Krinkle: unused memory is wasted memory [21:37:47] (03PS1) 10Krinkle: integration/monitoring: Add graph for last month [integration/docroot] - 10https://gerrit.wikimedia.org/r/165370 [21:37:57] (03CR) 10Krinkle: [C: 032] integration/monitoring: Add graph for last month [integration/docroot] - 10https://gerrit.wikimedia.org/r/165370 (owner: 10Krinkle) [21:37:59] (03Merged) 10jenkins-bot: integration/monitoring: Add graph for last month [integration/docroot] - 10https://gerrit.wikimedia.org/r/165370 (owner: 10Krinkle) [22:20:25] Project beta-update-databases-eqiad build #4588: FAILURE in 24 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4588/ [22:24:15] 3Wikimedia Labs / 3deployment-prep (beta): Beta: Cannot save any page "DB connection error: Can't connect to MySQL server on '10.68.16.193' (4) (10.68.16.193)" - 10https://bugzilla.wikimedia.org/71764#c2 (10Krinkle) a:3None https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/4588/ https:/... [22:24:44] 3Wikimedia Labs / 3deployment-prep (beta): Beta: Cannot save any page "DB connection error: Can't connect to MySQL server on '10.68.16.193' (4) (10.68.16.193)" - 10https://bugzilla.wikimedia.org/71764 (10Krinkle) p:5Unprio>3Normal s:5normal>3critic [22:26:57] Reedy: can you take a look at ^^ [22:29:09] (03PS5) 10Krinkle: qunit-cleanup: rm -f when deleting file [integration/jenkins-job-builder-config] - 10https://gerrit.wikimedia.org/r/163848 (owner: 10Hashar) [23:45:04] Project beta-scap-eqiad build #24633: FAILURE in 1 minuto 9 segundos: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/24633/ [23:46:16] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #206: FAILURE in 2 minutos 53 segundos: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/206/ [23:46:53] Project browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #155: FAILURE in 35 segundos: https://integration.wikimedia.org/ci/job/browsertests-Math-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/155/ [23:55:14] Yippee, build fixed! [23:55:14] Project beta-scap-eqiad build #24634: FIXED in 1 minuto 16 segundos: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/24634/