[00:32:44] 3Continuous-Integration, MobileFrontend-General-or-Unknown: mwext-MobileFrontend-npm is failing to complete on several commits (not verifying commits, can't merge) - https://phabricator.wikimedia.org/T76354#824121 (10Krinkle) 5Open>3Resolved Let me know if this breaks again. [01:57:31] YuviPanda: I've been getting them for the past 2 months. Are you referring to a more recent one, or those? [01:58:19] YuviPanda: Cool. That's good to hear. Got a link for me? I'm curious what it was in this case. aptitude? [02:19:46] Yippee, build fixed! [02:19:47] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #331: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/331/ [02:30:18] Yippee, build fixed! [02:30:18] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #349: FIXED in 1 hr 23 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/349/ [02:41:43] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #325: FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/325/ [03:29:00] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<10.00%) [03:54:04] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<10.00%) [04:48:58] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<10.00%) [05:35:16] Project beta-scap-eqiad build #32941: FAILURE in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32941/ [05:38:59] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<50.00%) [05:55:11] Yippee, build fixed! [05:55:12] Project beta-scap-eqiad build #32943: FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32943/ [06:39:02] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:48:37] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [09:17:32] PROBLEM - Puppet failure on deployment-sca-cache01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:18:39] RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:37:31] RECOVERY - Puppet failure on deployment-sca-cache01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:59:45] 3Continuous-Integration: Zuul: Implement support for customizing status_url to include the change.id - https://phabricator.wikimedia.org/T65744#824460 (10hashar) Sorry I got confused. This task would be fixed by https://review.openstack.org/#/c/86900/ which I have rebased and fixed. The other change (merged a... [10:04:12] 3Beta-Cluster: Puppet keep restarting jobrunner service - https://phabricator.wikimedia.org/T76999 (10hashar) 3NEW p:3Triage [10:04:31] 3Beta-Cluster: Puppet keep restarting jobrunner service - https://phabricator.wikimedia.org/T76999#824466 (10hashar) [10:04:59] 3Beta-Cluster: Puppet keep restarting jobrunner service - https://phabricator.wikimedia.org/T76999#824466 (10hashar) [10:25:07] 3Beta-Cluster: Puppet keeps restarting jobrunner service - https://phabricator.wikimedia.org/T76999#824518 (10hashar) [10:28:27] 3Continuous-Integration: zuul-cloner fails with "AttributeError: 'IterableList' object has no attribute 'origin' " (operations-apache-config-lint) - https://phabricator.wikimedia.org/T76955#824525 (10hashar) p:5Triage>3Normal a:3hashar [10:29:13] 3Continuous-Integration: zuul-cloner fails with "AttributeError: 'IterableList' object has no attribute 'origin' " (operations-apache-config-lint) - https://phabricator.wikimedia.org/T76955#823552 (10hashar) The git repository does not have a remote anymore :-/ [10:29:34] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #331: FAILURE in 5 min 41 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/331/ [10:35:09] 3Continuous-Integration: zuul-cloner fails with "AttributeError: 'IterableList' object has no attribute 'origin' " (operations-apache-config-lint) - https://phabricator.wikimedia.org/T76955#824533 (10hashar) The job has been aborted while cloning the repository: ``` 00:00:05.305 INFO:zuul.Cloner:Creating repo o... [10:35:37] 3Continuous-Integration: zuul-cloner fails with "AttributeError: 'IterableList' object has no attribute 'origin' " (operations-apache-config-lint) - https://phabricator.wikimedia.org/T76955#824536 (10hashar) [10:36:54] 3Continuous-Integration: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#824542 (10hashar) >>! In T72068#823563, @Se4598 wrote: > FYI: Maybe T76955 is related or even blocks this task. Yup some oddity in zuul-cloner caused the Jenkins job workspac... [10:46:48] 3Continuous-Integration: Jenkins: Re-enable lint checks for Apache config in operations-puppet - https://phabricator.wikimedia.org/T72068#824562 (10hashar) [10:52:10] Krinkle|detached: yesterday’s were caused by a tiny oversight, fixed in https://gerrit.wikimedia.org/r/#/c/178133/ [10:52:46] Krinkle|detached: outside of that, yes, apt get is a leading cause https://phabricator.wikimedia.org/T76771 [10:53:06] Krinkle|detached: and sometimes git / gerrit too - if you have a git clone and it 503s (happens sometimes!), puppet fails again [11:02:25] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #332: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/332/ [11:24:25] Yippee, build fixed! [11:24:25] Project browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #326: FIXED in 21 min: https://integration.wikimedia.org/ci/job/browsertests-UniversalLanguageSelector-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/326/ [11:38:42] Project browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce build #350: FAILURE in 1 hr 27 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-firefox-sauce/350/ [12:14:15] Project beta-scap-eqiad build #32978: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32978/ [12:23:02] Yippee, build fixed! [12:23:03] Project beta-scap-eqiad build #32979: FIXED in 7 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/32979/ [13:17:35] (03PS3) 10Tobias Gritschacher: Add jobs for WikibaseJavaScriptApi [integration/config] - 10https://gerrit.wikimedia.org/r/176232 (owner: 10Adrian Lang) [14:27:37] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#824912 (10zeljkofilipin) >>! In T75898#823816, @Stan3 wrote: > so I should rebase on env-abstraction-layer? @stan3: sorry, I do not rebasing will be possible. The env-abstraction-layer branch is a rewrite, as far... [14:56:15] WTF guys [14:56:16] https://integration.wikimedia.org/ci/job/mwext-UploadWizard-testextension/246/console [14:56:30] 14:54:28 zuul-cloner: error: Can not mix change and refupdate parameters [14:56:53] hashar: Are you working on testextension jobs or zuul-merger? [14:57:01] https://integration.wikimedia.org/ci/job/mwext-AJAXPoll-testextension/4/console [14:57:14] Just started failing randomly [14:57:18] Gasp, hashar is back! <3 [14:57:39] * Krinkle mumbles about testing in production :P [14:58:12] Krinkle: yeah that is me sorry [15:00:24] Krinkle: I have pushed some wrong changes by mistake :/ [15:04:02] hashar: mediawiki-phpunit is affected as well in extensions and core. [16:31:43] hi zeljkof are you here? I'd like to ask a question about Cucumber style, but it can wait if you're busy [16:31:52] chrismcmahon: sure, go ahead [16:33:16] zeljkof: should lines 103-113 be removed and have those values moved directly to the steps in Feature? https://gerrit.wikimedia.org/r/#/c/178205/2/tests/browser/features/step_definitions/mmv_download_steps.rb [16:33:54] zeljkof: in the past I have removed steps that did nothing but call one other step [16:34:50] chrismcmahon: just a second to finish something [16:47:17] 3Release-Engineering, Continuous-Integration: Zuul-cloner forgets to clear workspace - https://phabricator.wikimedia.org/T76304#827054 (10Krinkle) [16:48:39] hashar: Regarding https://phabricator.wikimedia.org/T76304, that has to be resolved soon. Otherwise I don't see another option but to remove zuul-cloner from jobs I care about. So far It's only provided theoretical advantages, minor speed ups or for important future plans. But in the here and now it's breaking stuff and not compatible with how git/jenkins work. [16:52:36] 3Release-Engineering, Continuous-Integration: Zuul-cloner forgets to clear workspace - https://phabricator.wikimedia.org/T76304#827240 (10Krinkle) Other work arounds I can imagine: Inside Zuul-cloner, run `git clean -dffx` in each extension, and then run it dry in core, filter out extensions from the list and ma... [16:59:24] chrismcmahonbrb: sorry for the delay, will take a look now [17:06:22] chrismcmahonbrb: the steps look fine to me [17:06:32] I would not remove them [17:38:37] PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:46:12] James_F and ryasmeen|Away: you might know: is this a feature or a bug in VE? as of Friday, there is an invisible set of "ghost" textareas in the Cite UI. https://gerrit.wikimedia.org/r/#/c/178218/1/modules/ve-mw/tests/browser/features/support/pages/visual_editor_page.rb [17:46:52] IOW every other textarea in the Cite UI is invisible, and seems to have no function [17:47:00] chrismcmahon: Is that the ULS stuff? [17:47:09] * James_F doesn't know. [17:47:57] hmmm [17:49:09] James_F: I'm going to merge that for the sake of green builds and sort it out later (or never) [17:49:17] * James_F nods. [17:50:43] chrismcmahon: feature [17:51:00] we use it for magic stuff [17:51:29] chrismcmahon: caused by https://gerrit.wikimedia.org/r/176476 [17:57:30] thanks MatmaRex (I am always a little suspicious of magic) [18:03:08] "feature" "magic stuff" ? scary :) [18:08:38] RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0] [18:11:33] greg-g: yeah, figuring out that failure was an interesting exercise. I might use it as a teachable moment later on today. [18:15:30] Project beta-scap-eqiad build #33017: FAILURE in 1 min 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33017/ [18:25:16] Yippee, build fixed! [18:25:17] Project browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce build #213: FIXED in 19 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-test2.wikipedia.org-linux-chrome-sauce/213/ [18:25:44] 3Continuous-Integration: [OPS] Jenkins: Slaves running Ubuntu Trusty should have hhvm installed - https://phabricator.wikimedia.org/T75356#830073 (10greg) p:5Triage>3High [18:25:51] 3Continuous-Integration: [upstream] Jenkins: jsduck test is sometimes passing when the build contains warnings - https://phabricator.wikimedia.org/T57668#830079 (10Chad) [18:26:11] 3Continuous-Integration: jenkins on release branch has problem when patch set contains js files - https://phabricator.wikimedia.org/T67085#830095 (10Chad) [18:37:09] Yippee, build fixed! [18:37:09] Project beta-scap-eqiad build #33019: FIXED in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33019/ [18:44:27] greg-g: chrismcmalunch btw, me and andrewbogott are tracking down ‘transient’ puppet failures across labs, to make shinken less spammy for you guys. [19:01:14] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #177: FAILURE in 34 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/177/ [19:02:50] YuviPanda: sweet [19:18:21] 3Quality-Assurance: rubocop fixes in mediawiki/selenium - https://phabricator.wikimedia.org/T75898#830827 (10dduvall) I'm hoping to get env-abstraction-layer merged this week, after which we can revisit the changes and see whether it would make more sense to rebase or start fresh. [19:26:58] 3Wikimedia-Logstash, Release-Engineering, Beta-Cluster: Make logstash in beta public - https://phabricator.wikimedia.org/T76784#831046 (10greg) [19:28:34] 3Release-Engineering, Beta-Cluster: Make Privacy Policy/ToS on Beta Cluster link to the labs version (not production version) - https://phabricator.wikimedia.org/T77858 (10greg) 3NEW p:3Normal [19:28:35] Hey, would anyone object to enabling CORS on betalabs? I'm trying to start work on a gadget that uploads files to betacommons from betaenwiki [19:47:08] 3Continuous-Integration: Zuul: Implement support for customizing status_url to include the change.id - https://phabricator.wikimedia.org/T65744#831396 (10hashar) I might be able to cherry-pick that patch on our production setup. I attempted it earlier this afternoon but pushed some wrong reference which caused a... [20:00:06] Yippee, build fixed! [20:00:06] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #403: FIXED in 1 hr 4 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/403/ [20:12:56] marxarelli: got time for a specific code review question? should lines 103-113 be removed in favor of using the lines they reference with values specified in the Feature? https://gerrit.wikimedia.org/r/#/c/178205/3/tests/browser/features/step_definitions/mmv_download_steps.rb [20:15:26] chrismcmahon: hmm, that's an interesting question [20:15:57] marxarelli: I think a step that does nothing but call a different step is a smell. [20:17:26] marxarelli: but Zeljko thought it was OK :-) I think I might make the change anyway in the name of simpler code [20:17:34] chrismcmahon: i generally agree that with that, but if we consider the feature to be a user story and less of a hard specification i think 'small'/'medium' makes more sense over exact pixel sizes [20:18:08] marxarelli: yeah, I thought of that. OTOH, if the test fails it obscures the reason for the failure [20:19:33] chrismcmahon: yes, that's true. i'd say 1) the more broad language in the feature makes sense, but 2) the implementation would be clearer without the added indirection (calling a step from a step) [20:31:39] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #232: FAILURE in 15 min: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/232/ [20:37:42] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #162: FAILURE in 59 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/162/ [21:10:19] marxarelli: updateded: I think it's better with those lines gone: https://gerrit.wikimedia.org/r/#/c/178205/ [21:20:53] 3Release-Engineering: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#831639 (10Qgil) 5Open>3Resolved Conclusion to the initial request: sure, please create quarterly milestones for your team as you wish. Just use the 'Sprint' label style and remember https://www... [21:34:19] Project beta-scap-eqiad build #33035: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33035/ [21:38:16] Yippee, build fixed! [21:38:16] Project beta-scap-eqiad build #33036: FIXED in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/33036/ [21:53:30] Yippee, build fixed! [21:53:30] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #456: FIXED in 1 hr 15 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/456/ [22:05:47] chrismcmahon: hmm ... now you have specific pixel sizes (implementation details) in both the feature and the steps [22:09:11] chrismcmahon: if they're important enough to the overall acceptance of the feature (ultimately up to the mmv folks) they should _all_ be in the .feature. if not, they should probably be left in the step definitions. but definitely don't put them in both places [22:09:40] marxarelli: looking... [22:11:58] marxarelli: aha. scope creep. I'll go there, one moment... [22:15:52] chrismcmahon: if i had to choose, i would say just leave it how it was :) [22:16:33] chrismcmahon: in other words, i vote for clarity in the .feature file over clarity in the step definition [22:16:57] marxarelli: I agree, but I think this gets us there, just a moment... [22:27:26] marxarelli: gah, I refactor this and I think I found a bug :-/ [22:27:31] looking... [22:46:04] marxarelli: not a bug, just a confusion [23:05:16] marxarelli: this is much more lean, and it still passes nicely: https://gerrit.wikimedia.org/r/#/c/178205/ [23:07:16] marxarelli: and FWIW, those size strings specified in the Features are presented to the user in UI, so I think it's legit to have them in Features [23:07:48] chrismcmahon: ah, that's a fair point [23:23:54] PROBLEM - Free space - all mounts on deployment-mediawiki01 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki01.diskspace._var.byte_percentfree.value (<100.00%) [23:38:42] can someone fix ^ before the instance dies? [23:41:19] yeah, change our log retention policy :/ [23:42:15] YuviPanda: I'll prune some crap there [23:42:47] bd808: I’m going to merge a change that’ll let you configure acct log retention more finely (currently 7 days), but atop is more problematic [23:43:00] why's that? (curious) [23:43:14] greg-g: atop uses very different log pruning mechanisms in precise and trusty :( [23:43:25] %$@#%*!&%^)!*^%@!*^%!*)^!@%# [23:43:28] greg-g: in precise it does a normal logrotate (easy enough) but in trusty uses a… custom cron job [23:43:36] !log Ran `apt-get clean` on deployment-mediawiki01 [23:43:38] Logged the message, Master [23:44:15] greg-g: so I need to override two things with a distbranch. will get to it this week sometime. [23:44:31] bd808: how much did that give us? [23:44:44] my core cleaning script isn't getting all cores anymore :( [23:45:08] !log deleted hhvm core on mediawiki01 [23:45:10] Logged the message, Master [23:45:11] bd808: you can set the core dump location via hiera now. you can set it to somewhere on NFS if you want? [23:45:26] Like /dev/null? ;) [23:45:33] https://wikitech.wikimedia.org/wiki/Hiera:Tools [23:45:38] bd808: heh, don’t think that actually works :) [23:45:46] 51% free now [23:45:57] bd808: haven’t found a reliable way to just… disable core dumps on all instances. [23:45:57] thanks man [23:46:08] well, ostensibly they're useful [23:46:33] The core file was ~500M [23:46:40] * greg-g nods [23:46:43] on a 2T partition :/ [23:46:51] g* [23:46:57] yeah 2G [23:47:07] too small for my brin to type it [23:47:11] *brain [23:47:14] E_DOESNOTCOMPUTE [23:47:25] E_ADDAUSBSTICK [23:47:29] haha [23:47:30] well, at least with new instances /var/log is resizeable [23:47:36] and /var is 2G and /var/log is 2G [23:47:41] and you can resize as you wish [23:47:44] what on earth does anyone do with a 500M core file. [23:47:49] god I just want to recreate all the instances.... [23:47:57] chrismcmahon: debug hhvm crashes [23:48:06] it's done it several times [23:48:08] greg-g: the mediawiki instances shouldn’t be too hard to recreate, I think. [23:48:14] once the new labs hardware is in place, at least. [23:48:36] o brave new world that has such core files in it and all that [23:48:45] all of beta should be easy to recreate. if it's not in puppet it doesn't count [23:49:33] and we’ve persistant puppet failures in only two hosts! [23:49:38] 500M is much better than a 6-8G jvm core file :) [23:49:47] 3Quality-Assurance, Multimedia, MediaWiki-extensions-MultimediaViewer: Automated screenshots - https://phabricator.wikimedia.org/T77634#832359 (10Gilles) [23:50:07] 2 too many [23:50:21] :/ [23:50:25] core files have gotten way bigger since the last time I had to load any into a debugger [23:50:44] moar ram == bigger cores mostly [23:51:07] The useful bits of the core file are generally pretty small [23:51:18] a couple of frames worth [23:53:55] RECOVERY - Free space - all mounts on deployment-mediawiki01 is OK: OK: All targets OK