[00:13:12] Project beta-scap-eqiad build #41303: FAILURE in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41303/ [00:22:00] Project beta-update-databases-eqiad build #7533: FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7533/ [00:22:33] Krinkle: Is Zuul/etc. stuck? Items listed with 100% 'success' but not closing. Lots of jobs "completed". [00:22:49] Oh, ha, belay that, it seems to be fixing itself suddenly. [00:35:01] Yippee, build fixed! [00:35:02] Project beta-scap-eqiad build #41306: FIXED in 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41306/ [01:21:31] Yippee, build fixed! [01:21:31] Project beta-update-databases-eqiad build #7534: FIXED in 1 min 30 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7534/ [01:45:09] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [03:09:43] Whom to poke for service deployment issues(s)? [03:09:48] greg-g: ^^ [03:15:19] what is the issue exactly? [03:25:19] 3Continuous-Integration: Consider running tmpreaper on Jenkins jobs' tmpfs - https://phabricator.wikimedia.org/T89327#1033479 (10Krinkle) p:5Triage>3High a:3Krinkle [03:26:06] 3Continuous-Integration: Consider running tmpreaper on Jenkins jobs' tmpfs - https://phabricator.wikimedia.org/T89327#1033472 (10Krinkle) Due to a security vulnerability not relevant to us (whereby an attacker with access to a non-root account can get root through setuid, SIGSTOP/SIGCONT, and patience), Debian d... [03:30:04] Krenair: submodule for cxserver is not updated. [03:30:27] Krenair: see: sca1001:/srv/deployment/cxserver/deploy/src [03:30:38] in beta? [03:30:47] Production [03:30:59] see in tin, it is uptodated. [03:31:04] ah, can't help you there, sorry [03:31:17] Krenair: no problem :) [03:44:10] Yippee, build fixed! [03:44:11] Project browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #447: FIXED in 1 min 12 sec: https://integration.wikimedia.org/ci/job/browsertests-CirrusSearch-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/447/ [03:45:28] 3Release-Engineering, ContentTranslation-cxserver, ContentTranslation-Deployments, Deployment-Systems: git deploy doesn't update submodule - https://phabricator.wikimedia.org/T89328#1033491 (10KartikMistry) 3NEW [03:46:52] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1033498 (10Krinkle) During a build, the tmpfs looks like this: ``` $ l total 1.3M drwxrwxrwx 3 jenkins-deploy wikidev 100 Feb 12 03:43 ./ drwxrwxrwt 1... [03:50:03] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #463: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/463/ [04:14:35] kart_: gwicke has the best first hand knowledge, otherwise alex [04:17:02] Yippee, build fixed! [04:17:02] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #478: FIXED in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/478/ [04:25:12] Yippee, build fixed! [04:25:13] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #308: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/308/ [04:43:58] greg-g: yep. Thanks. Had bug and mail to engg [04:48:29] 3Release-Engineering, ContentTranslation-cxserver, ContentTranslation-Deployments, Deployment-Systems: git deploy doesn't update submodule - https://phabricator.wikimedia.org/T89328#1033524 (10GWicke) Trebuchet and submodules are known to be problematic. One thing to double-check is the setup described in https:... [04:59:30] 3Release-Engineering, ContentTranslation-cxserver, ContentTranslation-Deployments, Deployment-Systems: git deploy doesn't update submodule - https://phabricator.wikimedia.org/T89328#1033527 (10KartikMistry) 'checkout_submodules' => true, is set for cxserver/deploy in manifests/role/deployment.pp [05:00:13] gwicke: it is fine to update submodule on individual hosts? [05:10:31] 3Release-Engineering, ContentTranslation-cxserver, ContentTranslation-Deployments, Deployment-Systems: git deploy doesn't update submodule - https://phabricator.wikimedia.org/T89328#1033531 (10KartikMistry) Another note: ``` kartik@sca1002:/srv/deployment/cxserver/deploy$ git status HEAD detached at cxserver/d... [05:28:37] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1033533 (10Krinkle) I've ruled out permissions as a possible cause. Digging further in the logs, build logs for failing jobs tend to be much more interlea... [05:30:11] James_F|Away: ^ [05:37:57] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #463: FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/463/ [05:45:51] kart_: afaik yes [05:46:36] kart_: IIRC the last time I did this I ran something with salt on the nodes itself to check out some pillar [05:46:50] gwicke: Strange thing here now. [05:46:52] [deploy] repo-name = cxserver/deploy checkout-submodules = false [05:47:03] in tin's deploy/.git/config [05:47:17] is the submodule flag set in puppet? [05:47:24] despite we've submodule true in puppet. [05:48:14] trebuchet continues to amaze me with the variety of failure modes it exhibits [05:48:21] :) [05:48:58] I guess you could try to manually change .git/config [05:49:07] https://git.wikimedia.org/blob/operations%2Fpuppet.git/29d75a6f5aeb8a4a442efb60ea6e21bce63cffde/manifests%2Frole%2Fdeployment.pp [05:49:18] gwicke: and deploy again? [05:49:39] yeah, but.. [05:49:50] just remembered that there is a similar flag somewhere on the clients [05:49:53] that got me last time [05:50:07] which of course isn't updated reliably either [05:50:47] ah [05:51:30] maybe .git/config on the clients as well? [05:52:04] https://wikitech.wikimedia.org/wiki/Trebuchet#Troubleshooting [05:52:25] the salt-run deploy.fetch 'test/testrepo' line sounds familiar [05:54:10] the problem with deployment systems is that nobody really owns them or is empowered to do something about it, and it's all related to everything else like containerization [06:14:40] 3Continuous-Integration: MediaWiki installs in Jenkins frequently fail to access their sqlite database due to locks - https://phabricator.wikimedia.org/T89180#1033549 (10tstarling) Passing array( PDO::ATTR_TIMEOUT => $timeout ) as the $options argument in the PDO constructor, with $timeout set to some sensible v... [06:16:06] gwicke: I think I don't have access to run salt-run on minions/sca [06:16:38] gwicke: if you're not bored and awake, can you do on sca1001/sca1002? [06:17:32] do you have sudo on sca1001? [06:17:58] gwicke: tried. nope. [06:18:14] what is the repo name? [06:18:20] cxserver/deploy? [06:18:23] yes [06:18:44] I don't have sudo either [06:19:18] sorry ;( [06:19:42] 3Release-Engineering, ContentTranslation-cxserver, ContentTranslation-Deployments, Deployment-Systems: git deploy doesn't update submodule - https://phabricator.wikimedia.org/T89328#1033563 (10KartikMistry) And, On tin, .git/config says ``` [deploy] repo-name = cxserver/deploy checkout-submodules = false ``` W... [06:19:54] :/ [06:20:04] if it makes you feel better, when I tried to test the entire restbase puppetization including trebuchet in labs I wasted almost two days on it, and it still didn't work reliably [06:20:13] that's why I wrote that doc page on wikitech [06:20:14] We need alex then. [06:20:41] yup [06:21:00] gwicke: No worries. Thanks a lot for help :) [06:22:20] there is a submodule listed in http://tin.eqiad.wmnet/cxserver/deploy/.git/modules/src [06:22:36] eh, /srv/deployment/cxserver/deploy/.git/config [06:31:11] gwicke: manual update of it and then salt-run is the way? or will salt-run fix that? [06:35:12] RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK [06:40:45] kart_: I think at some point the manual update helped, but last time I ran into the issue that there is some state on the client too [06:41:41] you could try nuking the deploy repo on tin & see if the setup does the right thing the second time around [06:42:21] or dive into trebuchet and salt code & try to figure it out [06:42:32] blah. I fear that :/ [06:42:41] salt access anyway need alex. [06:42:56] *nod* [06:43:58] back in 2004 we had a deploy system that was really fast [06:43:59] 3MediaWiki-extensions-MathSearch, Continuous-Integration: MathSearch tests fail - https://phabricator.wikimedia.org/T89237#1033578 (10Physikerwelt) @hashar: I totaly agree. Maybe a trigger that will be called on test coverage expansion and triggers all extension tests would be nice. [06:44:11] 1) you open a shell on the nfs server [06:44:18] 2) you type vim somefile.php [06:44:23] 3) you do some hacking [06:44:34] 4) :w and reload wikipedia.org [06:45:02] 5) if it doesn't work, quickly uuuu:w [06:45:03] ;) [06:51:02] 3Continuous-Integration: Create a trigger to run extension tests on test coverage extension - https://phabricator.wikimedia.org/T89333#1033583 (10Physikerwelt) 3NEW [07:00:39] gwicke: oh, that way! ;) [08:22:41] (03PS6) 10Hashar: zuul: Make operations-puppet-puppetlint-lenient voting [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (https://phabricator.wikimedia.org/T87132) (owner: 10Matanya) [08:24:25] (03CR) 10Hashar: [C: 032] "Per discussion on OPS list." [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (https://phabricator.wikimedia.org/T87132) (owner: 10Matanya) [08:24:37] \o/ [08:24:59] matanya: thanks a lot for that patch :] [08:25:11] I love it when people take care of configuring the CI [08:25:19] (03Merged) 10jenkins-bot: zuul: Make operations-puppet-puppetlint-lenient voting [integration/config] - 10https://gerrit.wikimedia.org/r/189589 (https://phabricator.wikimedia.org/T87132) (owner: 10Matanya) [08:25:33] at your service hashar :) [08:45:34] committing from public coffee to coworking place [08:45:38] comutte [08:45:40] bah [08:45:45] english crap. [08:46:01] That's why we had/has French Jenkins. [08:46:04] :D [08:52:29] 3Continuous-Integration, ContentTranslation-Deployments, MediaWiki-extensions-ContentTranslation: Enable Debian CI tests on all Apertium packages - https://phabricator.wikimedia.org/T87607#1033681 (10KartikMistry) @hashar Thanks! [08:55:06] 3MediaWiki-Unit-tests, Continuous-Integration: TextPassDumperTest::testCheckpointGzip expected more than 1 checkpoint to have been created - https://phabricator.wikimedia.org/T70653#1033685 (10Krinkle) p:5Low>3High [08:55:21] 3MediaWiki-Unit-tests, Continuous-Integration: TextPassDumperTest::testCheckpointGzip expected more than 1 checkpoint to have been created - https://phabricator.wikimedia.org/T70653#738146 (10Krinkle) Now happening on mediawiki-core jobs, too. https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/2566... [09:52:13] PROBLEM - Puppet failure on deployment-pdf02 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [10:17:14] RECOVERY - Puppet failure on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:17:56] PROBLEM - Content Translation Server on deployment-cxserver03 is CRITICAL: Connection refused [10:19:28] 3pywikibot-core, Continuous-Integration: pip install --process-dependency-links fails on build boxes - https://phabricator.wikimedia.org/T89231#1033727 (10hashar) 5Open>3Resolved a:3hashar The errors were due to labs infrastructure not resolving DNS properly, that caused tox/pip to no more be able to fetch... [10:20:28] 3pywikibot-core, Continuous-Integration: pip install --process-dependency-links fails on build boxes - https://phabricator.wikimedia.org/T89231#1033730 (10hashar) p:5Triage>3High [10:20:54] (03PS4) 10Hashar: browsertests: move user/pass to Credentials plugin [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) [10:30:12] (03CR) 10Hashar: [C: 032] "I have updated all jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) (owner: 10Hashar) [10:37:03] (03Merged) 10jenkins-bot: browsertests: move user/pass to Credentials plugin [integration/config] - 10https://gerrit.wikimedia.org/r/189938 (https://phabricator.wikimedia.org/T89226) (owner: 10Hashar) [10:37:55] RECOVERY - Content Translation Server on deployment-cxserver03 is OK: HTTP OK: HTTP/1.1 200 OK - 1103 bytes in 0.022 second response time [10:41:54] !log Removing MEDIAWIKI_PASSWORD* global env variables from Jenkins configuration {{bug|T89226}} [10:41:57] Logged the message, Master [11:11:30] !log changed passwords of selenium users. [11:11:35] Logged the message, Master [11:40:32] zeljkof: pull request of what we just did: https://github.com/amire80/screenshot/pull/2 [11:53:31] 3Release-Engineering, translatewiki.net: Browser test for sandbox.translatewiki.net fails because of a SSL host mismatch (cert has 'translatewiki.net') - https://phabricator.wikimedia.org/T89244#1033918 (10hashar) a:3Amire80 [11:54:57] 3Release-Engineering, translatewiki.net: Browser test for sandbox.translatewiki.net fails because of a SSL host mismatch (cert has 'translatewiki.net') - https://phabricator.wikimedia.org/T89244#1031076 (10hashar) p:5Triage>3Normal [11:55:43] 3Release-Engineering, Parsoid, Architecture, Services: Distribution strategy option: Use Debian packages - https://phabricator.wikimedia.org/T88154#1033935 (10hashar) p:5Triage>3Low [12:25:32] zeljkof: pull request of what we did earlier today: https://github.com/amire80/screenshot/pull/2 * you asked to poke you for review :) [12:34:47] Project beta-scap-eqiad build #41378: FAILURE in 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41378/ [12:41:46] aharoni: reviewed both the gem and VE commits [12:46:50] hello zeljkof! [12:54:50] Yippee, build fixed! [12:54:50] Project beta-scap-eqiad build #41380: FIXED in 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41380/ [13:00:08] hi vikasyaligar [13:00:13] what's up? :) [13:06:54] zeljkof: thank you for reviewing ! :) [13:07:46] vikasyaligar: no problem, sorry for the delay, I was traveling, still catching up [13:16:12] zeljkof: I am there finally [13:16:19] went out for lunch to late [13:16:45] hashar: no problem, I need a few minutes to finish something [13:16:48] are you ready? [13:16:55] in hangout [13:22:20] hashar: joining [13:55:13] zeljkof: https://integration.wikimedia.org/ci/job/operations-puppet-tox-data_admin_lint/12574/ \O/ [13:55:29] zeljkof: https://integration.wikimedia.org/ci/job/operations-puppet-tox-data_admin_lint/12574/ [13:55:33] grr [13:55:36] https://integration.wikimedia.org/zuul/ [13:55:41] zeljkof: https://integration.wikimedia.org/zuul/ [14:02:58] 3Ops-Access-Requests, Continuous-Integration: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#1034144 (10akosiaris) [14:32:42] 3Project-Creators, Wikimedia-Labs-wikistats, MediaWiki-extensions-OpenStackManager, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-wikitech-interface, Labs, Labs-Vagrant, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-Other, Beta-Cluster, Wikimedia-Labs-extdist, Wikimedia-Labs-General: Labs' Phabricator... [14:43:33] 3Release-Engineering: Ask Sauce Labs support if there is a way to disable Selenium log temporarily - https://phabricator.wikimedia.org/T89353#1034234 (10zeljkofilipin) 3NEW a:3zeljkofilipin [14:43:44] hashar: ^ [14:47:46] zeljkof: https://gerrit.wikimedia.org/r/#/c/190210/1/modules/admin/data/data.yaml [14:59:15] hi zeljkof did someone change the password for Selenium_user on beta labs? [14:59:22] hashar? ^^ [14:59:32] chrismcmahon: yes, hashar didi [14:59:34] did [14:59:43] just a sec, will send you the new passwords [14:59:57] chrismcmahon: yeah I have changed them all [15:00:03] forgot to mail foks [15:00:04] folks [15:00:05] chrismcmahon: https://office.wikimedia.org/wiki/Selenium_passwords [15:00:12] they are on office wiki [15:00:15] aharoni: https://office.wikimedia.org/wiki/Selenium_passwords [15:00:15] not to be changed though [15:00:18] great, thanks [15:00:19] since eJenkins has a copy as well [15:00:46] 3Release-Engineering, ContentTranslation-cxserver, ContentTranslation-Deployments, Deployment-Systems: git deploy doesn't update submodule - https://phabricator.wikimedia.org/T89328#1034258 (10Ottomata) 5Open>3Resolved a:3Ottomata For now, I think I fixed this, although Alex says there is more to it. If... [15:01:41] chrismcmahon: the password in Jenkins are now stored in a credential store https://integration.wikimedia.org/ci/credential-store/domain/browsertests/ [15:05:44] is there any reason the new password would not work for use with the API? [15:06:16] * chrismcmahon double checks [15:07:30] my bad, user error [15:08:32] so it seems like iframes are finally gone from VE [15:13:32] zeljkof: sudo -su zuul [15:20:11] JimConsultant: dear consultant. Isn't it ironic we are talking about triaging mw logs and logstash is actually disabled for them ? :D [15:21:30] on a more serious note, what do you think of having an almost daily checkin here? (poke JimConsultant chrismcmahon ) [15:21:50] I now have 1/1 with zeljkof three times per week. Turns out to be useful [15:21:59] I don't think logstash should be used for anything right now, it's too sick [15:22:29] why did it get disabled anyway ? [15:22:39] was it due to the elastic search backend being overloaded? [15:23:24] MW logs to logstash was part of what made last week's outage longer than it needed to be [15:23:27] Because redis [15:25:16] pff [15:25:20] too many levels of inceptions [15:25:28] the redis queue went unresponsive? [15:25:39] Yep [15:28:45] https://wikitech.wikimedia.org/wiki/Incident_documentation/20150205-SiteOutage [15:30:13] goooooood morning thcipriani ! [15:30:33] and a good afternoon to you. [15:30:41] may I ask what kind of crazy assignment you have been appointed? [15:31:32] sure, my first task is to try to do some review (i.e., get it working locally, first) of the proposed solution to the logstash problem: https://gerrit.wikimedia.org/r/#/c/189672/1 [15:32:00] :-D [15:32:45] I have no clue how the logs are sent :-( [15:33:37] 3Wiki-Release-Team, Release-Engineering, Engineering-Community: Clean #Wiki-Release-Team project - https://phabricator.wikimedia.org/T88263#1034328 (10Aklapper) IMHO done here, only waiting for T88965 whether to reassociate or decline T532,T533,T536,T951,T1029,T1030,T1031,T1032,T1114 [15:33:44] really the past couple days, as far as development stuff goes, I've been trying to familiarize myself with vagrant/puppet (since I haven't worked with puppet in the past—mostly been at Chef shops) and digging through mediawiki/core, just trying to familiarize myself with request flow. I've got this change checked out...still trying to "test" it :) [15:35:36] is there an easy way to point cucumber tests running locally to saucelabs IE boxen? [15:38:48] 3Project-Creators, Wikimedia-Labs-wikistats, MediaWiki-extensions-OpenStackManager, Wikimedia-Labs-Infrastructure, Wikimedia-Labs-wikitech-interface, Labs, Labs-Vagrant, Tool-Labs-tools-Article-request, Tool-Labs, Wikimedia-Labs-Other, Beta-Cluster, Wikimedia-Labs-extdist, Wikimedia-Labs-General: Labs' Phabricator... [15:39:16] I guess that answers my question https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing/Running_tests#Running_browser_tests_at_SauceLabs [16:09:32] 3Release-Engineering, Parsoid, Architecture, Services: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#1034406 (10GWicke) p:5Triage>3High [16:11:02] (03Abandoned) 10Hashar: Reset & clean workspace repo for better hygiene [integration/zuul] - 10https://gerrit.wikimedia.org/r/185686 (https://phabricator.wikimedia.org/T76304) (owner: 10Ori.livneh) [16:20:15] 3Release-Engineering, Parsoid, Architecture, Services: Evaluate and decide on a distribution strategy targeted at VMs - https://phabricator.wikimedia.org/T87774#1034430 (10GWicke) I bumped this to high priority as I think that getting clarity on which direction we want to go in soon is important to let us make t... [16:22:47] * greg-g yawns [16:38:19] 3Multimedia, MediaWiki-extensions-UploadWizard, Quality-Assurance: UploadWizard browser test for chunked upload - https://phabricator.wikimedia.org/T89289#1034542 (10MarkTraceur) p:5Triage>3Normal [16:40:05] and I am off [16:40:07] will be back on monday [16:40:10] hopefully [16:46:02] JimConsultant, thcipriani: It would be really super cool and awesome to have https://gerrit.wikimedia.org/r/#/c/189672/ merged today so we can test on beta by changing the mw-config to use syslog instead of redis. [16:46:28] * bd808 has a mostly open day today to work on this before heading back into the land of being a PM [16:47:00] Let me finish trying to hunt down this CentralNotice thing and then sure. [16:47:58] *nod* [16:49:02] bd808: been looking at the revision a while. Still trying to figure out exactly _how_ to make this work with the vagrant instance. Lots of background knowledge to build up... [16:50:21] Apply https://gerrit.wikimedia.org/r/#/c/189677/ to mw-vagrant; vagrant roles enable elk; vagrant reload; visit http://logstash.local.wmftest.net:8080/ to see kibana. [16:50:27] Do I have to be using the elk role with vagrant? Seems to keep failing on provision. [16:50:28] sooooo simple ;) [16:50:57] the elk role likes to blow up when elasticsearch is first installed [16:51:12] lemme try reload. I thought I tried that, I think I've just been trying provision, but I could be wrong. [16:51:13] race for the jvm to start before things try to talk to it [16:52:12] Making that role (and the cirrussearch role) more stable on initial provision would be nice but I haven't poked at it much [16:53:49] 3Release-Engineering, Project-Creators: Add in Phabricator quarterly milestones for RelEng - https://phabricator.wikimedia.org/T75729#1034567 (10Aklapper) 5Open>3stalled Setting status to stalled as it looks like folks/teams need to discuss what they actually want (but maybe that discussion has taken place a... [16:55:10] Looks like vagrant reload --provision got it. The role kept failing when trying to curl...something. IIRC it was something with elasticsearch, but I guess I don't have that much scrollback. [16:56:06] Yeah. It was trying to setup config by sending curl messages to elasticsearch [16:56:38] which breaks if the elasticsearch jvm isn't live to http requests yet [16:57:20] I think my magic trick to force the vm to reload after enabling elasticsearch is broken these days too [16:57:31] it was a gross hack in the first place [16:58:43] mediawiki-vagrant has become a bit cumbersome. In trying to be all things to all people it has become few things to few people [16:59:09] I love it, but I wrote a ton of the mess and can readily read the erro stack to see what I need to do next [16:59:20] E_TOOMANYFEATURES [17:00:32] bd808: perhaps some basic integration/smoke tests will help (i have a handful already working) [17:01:20] given elasticsearch is enabled when i run vagrant provision then the shit don't blow up [17:02:39] *nod* cold rebuilds are our bane because most of us don't try that before merging roles. And there are some many combinations of things that can cause ordering problems. [17:02:51] This is a classic build automation problem really [17:03:18] incremental development works but fresh setup often doesn't [17:03:53] which is why we should all be more strict about blowing the vm away and starting from scratch more often (myself included) [17:04:14] right. i want to experiment with puppet-rspec as well, and see if it might have some value when it comes to ensuring proper puppet dependency chains, etc. [17:04:41] * marxarelli *nods* [17:04:48] i'm pretty bad about that [17:05:24] if you can figure out how to make valuable tests with it that would be swell. Most of the puppet-rspec tests I've read seem less than useful [17:05:56] such is testing i think, finding the happy median [17:06:03] true enough [17:06:27] highest value:debt ratio [17:06:53] I had a cowoerker who liked to do things like assert that new X produced an instanceof X [17:06:53] and if value/debt < 1, don't do it :) [17:07:15] haha [17:07:19] yeah, that's pointless [17:07:58] that's not a unit of your code, that's a unit of zend/hhvm [17:09:29] My boss at the time thought it was great. "Look at all the asserts from Bob's tests! He's so thorough!". I just went back to writing code that made the product work and kept my mouth shut [17:09:39] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1034610 (10greg) I'm going to go with @legoktm's suggestion of #Wikimedia-log-errors unless anyone vetoes me :) To address the concern of it being too big/unwieldy: h... [17:09:52] well, technically in php3, you could reassign `$this` in the constructor... [17:09:53] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1034614 (10greg) [17:09:59] nm. php4 [17:10:19] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1032707 (10greg) [17:16:52] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1034631 (10greg) [17:18:47] 3Release-Engineering: Make log responsibilities changes - https://phabricator.wikimedia.org/T89049#1034643 (10greg) [17:18:50] 3Release-Engineering, Project-Creators: Create project/tag to collect fatal/exception log related bugs - https://phabricator.wikimedia.org/T89292#1034641 (10greg) 5Open>3Resolved Donezors. [17:19:34] For God and Country: https://phabricator.wikimedia.org/tag/wikimedia-log-errors/ [17:20:02] is https://phabricator.wikimedia.org/tag/mediawiki-skins-blueprint/ a 404 for anyone else? [17:20:47] legoktm: yep, 404 [17:21:50] o.O [17:24:04] Same as Phab admin, it's really not there [17:24:21] but it's showing up in search and I see it on tasks and stuff...? [17:29:12] #blueprint redirects to that too [17:29:30] https://phabricator.wikimedia.org/tag/blueprint/ [17:29:46] That's a design project [17:31:26] It's hard to fight the community's perception of us doing things in private when we make private projects. [17:31:36] * JimConsultant sighs [17:31:52] I filed https://phabricator.wikimedia.org/T89361?workflow=create [17:32:53] There are a small handful of folks who just don't get how freeing transparency is. But they are also folks who get trolled pretty hard so ... correlation [17:52:55] bd808: so looking at c/189672—everything seems to be working: I can see the UDP packets on my vagrant box, sending valid json, and it's making it into my logstash, and I have no qualms with the implementation. What are my next steps? [17:53:23] push review and +2 :) [17:54:13] It will merge into master and be deployed to beta. Then I can cherry-pick my ops puppet changes there and push a config change to use the new setup [17:54:44] we can watch it there for a bit to see that things work in the real world [17:55:20] New to gerrit—gonna have to look at how to do that, but sounds good. Thanks! [17:55:29] Assuming I didn't mess up then I can make backport patches for prod and poke folks to get this rolled out early next week [18:00:01] JimConsultant, I don't think this is a private project [18:00:14] deleted? [18:00:23] private projects give you 403, tell you the policy enforced, etc. [18:00:29] this is just broken [18:00:57] it could be both of course, but... :) [18:10:24] thcipriani: oh, forgot to ask: do you have any dietary choices to keep in mind for your welcome lunch? [18:11:21] greg-g: nope. I'm occasionally a conscious eater, but you've caught me at an easy time. [18:11:45] thcipriani: as in you don't fall asleep in your soup? [18:11:53] but now you do? [18:12:09] sleep eating. It's a problem for me. [18:12:32] narcamasticate [18:13:04] ^ good heavy metal band name [18:20:17] haha [18:30:01] hi ryasmeen I am updating VE tests because there are no more iframes [18:49:14] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #536: FAILURE in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/536/ [18:54:50] Project beta-scap-eqiad build #41421: FAILURE in 44 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41421/ [18:55:07] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #363: FAILURE in 3 min 39 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/363/ [19:07:30] okay chrismcmahon, we did update for most of them earlier right? [19:09:56] ryasmeen: there was one actual change for the Cite test, and a couple more patches that removed elements and steps that were unused [19:10:21] ryasmeen: all finished now, but there is still some stuff left for the language_screenshot test people to clean up [19:11:41] cool chrismcmahon! [19:14:57] Yippee, build fixed! [19:14:58] Project beta-scap-eqiad build #41423: FIXED in 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/41423/ [19:15:12] 3Quality-Assurance: Update language_screenshot test - https://phabricator.wikimedia.org/T89370#1034845 (10Cmcmahon) 3NEW a:3Vikassy [19:15:50] Project browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce build #479: FAILURE in 13 min: https://integration.wikimedia.org/ci/job/browsertests-UploadWizard-commons.wikimedia.beta.wmflabs.org-linux-firefox-sauce/479/ [19:17:53] 3VisualEditor, Quality-Assurance: Update language_screenshot test - https://phabricator.wikimedia.org/T89370#1034845 (10Amire80) [19:20:06] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #293: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/293/ [19:21:28] 3operations, Continuous-Integration: [upstream] Create a Debian package for Zuul - https://phabricator.wikimedia.org/T48552#1034864 (10hashar) [19:25:55] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #502: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/502/ [19:26:05] Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #344: FAILURE in 2 min 56 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/344/ [19:28:53] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #309: FAILURE in 39 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/309/ [19:40:22] PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:45:27] btw, JimConsultant, good work on the centralnotice log stuff today [19:45:48] 3Release-Engineering: Move browser test alerts to responsible teams' channels from -releng - https://phabricator.wikimedia.org/T89375#1034923 (10greg) 3NEW [19:45:50] Thx. I hope they can get a fix out shortly [19:45:54] yeah [19:56:14] 3Continuous-Integration: jenkins fails for calling lint on folders with a space in its name - https://phabricator.wikimedia.org/T89380#1034975 (10Umherirrender) 3NEW [20:02:55] JimConsultant: is search on beta labs still sketchy? I'm seeing some unexpected results searching for media files. [20:03:22] Might be missing some results [20:03:26] But it should be functionally ok [20:03:32] (results will slowly catch up) [20:03:46] seems like it is missing some results. I won't panic then :-) [20:07:13] Yippee, build fixed! [20:07:14] Project browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #512: FIXED in 19 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/512/ [20:35:43] 3Continuous-Integration: jenkins fails for calling lint on folders with a space in its name - https://phabricator.wikimedia.org/T89380#1035072 (10hashar) The Jenkins script (in integration/jenkins.git) runs: /bin/git-changed-in-head php php5 inc phtml module install Which yields modified files matching file e... [20:40:21] I have to update the test, no wait, that's OK, and that's OK... oh, it's actually a bug. [21:13:57] marxarelli: we're scheduled to meet in 15 minutes but I'm not sure I have anything useful to talk about after our discussion with Zeljko this morning [21:14:18] chrismcmahon: yeah, me neither. want to just wait until next week's session? [21:14:32] marxarelli: I'm thinking maybe we should postpone in favor of next week's 3-way training session [21:14:45] chrismcmahon: sounds good [21:15:17] marxarelli: cool, I'll go back to railing at selenium, which for some reason won [21:15:39] won't type "&" into a contentEditable field all of a sudden, when it will for Chrome [21:17:01] Yippee, build fixed! [21:17:01] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #10: FIXED in 8 min 59 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/10/ [21:21:06] 3MediaWiki-extensions-SpamBlacklist, Continuous-Integration, MediaWiki-Core-Team: Figure out a system to override default settings when in test context - https://phabricator.wikimedia.org/T89096#1035188 (10Krinkle) Sorry for being pessimistic, but I'm challenging this to draw out more use cases. As it stands, I... [21:41:12] 3Continuous-Integration: jenkins fails for calling lint on folders with a space in its name - https://phabricator.wikimedia.org/T89380#1035313 (10Krinkle) Use the `-z` option. Many git commands (especially those meant for machine reading) support it. Including [git-show](http://git-scm.com/docs/git-show) as used... [21:51:35] 3Wikimedia-Fundraising, Quality-Assurance, Continuous-Integration: Create unit and integration tests for Fundraising extensions to identify breaking MediaWiki changes - https://phabricator.wikimedia.org/T89404#1035380 (10bd808) 3NEW [22:06:06] PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<12.50%) [22:15:34] 3Wikimedia-Fundraising, Quality-Assurance, Continuous-Integration: Create unit and integration tests for Fundraising extensions to identify breaking MediaWiki changes - https://phabricator.wikimedia.org/T89404#1035427 (10atgo) Adding fr-tech (feel free to unsub if you'd like) [22:51:14] (03PS1) 10Hashar: TwoFactorAuthentication now pass jshint [integration/config] - 10https://gerrit.wikimedia.org/r/190370 (https://phabricator.wikimedia.org/T63641) [22:51:27] (03CR) 10Hashar: [C: 032] TwoFactorAuthentication now pass jshint [integration/config] - 10https://gerrit.wikimedia.org/r/190370 (https://phabricator.wikimedia.org/T63641) (owner: 10Hashar) [22:52:29] (03Merged) 10jenkins-bot: TwoFactorAuthentication now pass jshint [integration/config] - 10https://gerrit.wikimedia.org/r/190370 (https://phabricator.wikimedia.org/T63641) (owner: 10Hashar) [22:53:28] 3Continuous-Integration: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#1035521 (10hashar) [23:04:06] (03PS2) 10Krinkle: Stop Zend tests on mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/187904 (owner: 10Hashar) [23:05:27] (03Abandoned) 10Sn1per: Enable jshint voting for TwoFactorAuthentication [integration/config] - 10https://gerrit.wikimedia.org/r/186141 (https://phabricator.wikimedia.org/T63641) (owner: 10Sn1per) [23:07:20] (03PS3) 10Krinkle: Stop Zend tests on mediawiki/core [integration/config] - 10https://gerrit.wikimedia.org/r/187904 (owner: 10Hashar) [23:08:10] (03PS4) 10Krinkle: Remove phpunit-zend from mediawiki-core test pipeline (kept in gate) [integration/config] - 10https://gerrit.wikimedia.org/r/187904 (owner: 10Hashar) [23:09:47] (03CR) 10Krinkle: [C: 031] "Seems good to go. Though it may be annoying to have to wait for the gate pipeline to find out a Zend incompatibility. 4 minutes isn't bad." [integration/config] - 10https://gerrit.wikimedia.org/r/187904 (owner: 10Hashar) [23:11:37] (03PS7) 10Krinkle: Clean up phpcs macros and jobs (remove strict/lenient split) [integration/config] - 10https://gerrit.wikimedia.org/r/166071 (https://phabricator.wikimedia.org/T50420) (owner: 10Jforrester) [23:19:04] marxarelli or anyone good with browser tests: is it possible to customize what happens on a visit() call? [23:19:23] I am wondering if it is possible to handle UploadWizard steps as separate pages [23:19:24] tgr: what do you mean by customize? [23:19:47] but there is no way to get to them via URL, you need to start at the first step and click through [23:20:27] tgr: you mean to start the test on different pages of the process for uploading? I never figured out how that would be possible. [23:20:47] so normally visit(somepage) just opens somepage.url [23:21:15] can I override some method to perform a complex sequence of browser commands instead? [23:21:24] chrismcmahon: yeah [23:21:30] tgr: if you want to encapsulate the actions necessary to get to a certain step of uploading, you could write your own page-object method [23:22:04] which browser commands exactly? selenium is designed to be capable only of what the user can do and see (with some exceptions, but the exceptions are not nice) [23:22:37] go to Special:UploadWizard, add some file, click next etc.etc. [23:23:08] I was just wondering if there was a way to do that via the visit() syntax [23:23:57] tgr: so, it might help to understand what visit() and on() do [23:24:19] tgr: one moment, maybe I have something... [23:24:30] on(SomePage) simply instantiates a new SomePage object with the current value of `@browser` [23:24:55] it's equivalent to `SomePage.new(@browser)` [23:25:23] visit() does the same thing, but calls `page.visit` as well [23:25:40] so it's equivalent to `page = SomePage.new(@browser); page.visit` [23:25:45] tgr: Cucumber allows you to re-use steps within a single step, for example https://github.com/wikimedia/mediawiki-extensions-VisualEditor/blob/master/modules/ve-mw/tests/browser/features/step_definitions/multiedit_workflow_steps.rb [23:26:24] chrismcmahon: sure, I just want to come up with a syntax that's easy to read [23:26:25] what this means for you, is that you can define your own methods of SomePage, and call them on the page object [23:26:43] marxarelli: so I just override .visit basically? [23:27:29] tgr marxarelli the problem with the UW UI is that we're required to navigate sequentially, and each subsequent step depends on information provided by the previous step. [23:27:30] so, `class UploadWizardPage; def complete_upto(step); # perform actions; end` [23:28:10] then, in your step definition, `page = visit(UploadWizardPage); page.complete_upto(1)` or something [23:29:25] the other option is to just create compound steps that delegate/build off of other, simpler steps [23:31:03] e.g. 'Given(/^I have completed steps 1 and 2$/) do; step "visit UW page"; step "choose file"; step "fill out blah"; end` [23:32:07] tgr: what marxarelli just said is probably the most convenient thing. (unless we want to re-think how the whole UI works, which I would not mind doing) [23:33:01] chrismcmahon: we rethought it already, reimplementing is another matter though :) [23:33:53] the long-term plan is to have each file work as a separate box which can be in the upload, describe etc. state completely independently of the other boxes [23:34:08] I can dig up the design docs if you are interested [23:34:40] tgr: I actually *am* interested. no hurry, though. is it on mw.o? [23:40:12] chrismcmahon: yes, https://www.mediawiki.org/wiki/UploadWizard#New_Designs [23:40:29] not the one I had in mind, but the only one I can find [23:41:48] the point, which is maybe not that clear from those slides, is that each file would behave like a small wizard: you could go from one step to the other for that one file and it would not affect the others [23:42:03] thanks tgr. UW was one of the earliest browser tests, it was our first big challenge. [23:42:54] that makes a lot of sense. the page-after-page workflow that depends on state with a good chance of failing before the end is not very nice [23:43:43] marxarelli: let me start again, I don't think I made sense :) [23:44:22] the way UW works, you add files in step 1, assign a license to them in step 2, add descriptions in step3 [23:44:33] this is JS-based and all happens on the same page [23:45:47] so I could do visit(UWPage) do; add(someFile); add(someLicense); add(someDescriptions); ...test... [23:46:00] it's just not very nice to read [23:47:04] so I was wondering if there is a way to define a test file (with license data and all) in a config file, and then just say visit(UWPage(testfile).step2) [23:47:16] or something along those lines [23:47:41] it seems like it's possible if I rewrite the visit method [23:47:56] unless you think that's a horrible idea :) [23:48:03] tgr: what do you want to accomplish by doing that? [23:48:22] having short and easy-to-read step definitions [23:48:24] you don't need to rewrite visit(), just write your own method! :) [23:48:42] and extracting test uploadfile description to a single place [23:50:21] tgr marxarelli there is a lot of duplication in the UW test, but it seems simpler just to use Cucumber and make the Scenario have variable data at line 82 https://github.com/wikimedia/mediawiki-extensions-UploadWizard/blob/master/tests/browser/features/upload_wizard.feature [23:50:46] either a new method of UWPage (e.g. `visit(UWPage).upload(testFile, step: 2)` or something), or put it in a cucumber helper module. i can show you an example of the latter if it would help [23:51:57] marxarelli: I would take a look, if you have it at hand [23:53:38] tgr: https://gerrit.wikimedia.org/r/#/c/180324/2/tests/browser/features/support/flow_helper.rb [23:53:52] tgr: and https://gerrit.wikimedia.org/r/#/c/180324/2/tests/browser/features/support/env.rb [23:54:37] it's an example of how you can extend Cucumber's World environment to include methods specific to your own test cases [23:55:25] the general cucumber documentation is here https://github.com/cucumber/cucumber/wiki/A-Whole-New-World [23:56:42] a module is like a trait/mixin? [23:56:47] the first option that i mentioned simply requires you going into the page-object class and defining a new method with `def` (e.g. `def upload(file, step: 1)`) [23:57:11] tgr: that's one function of them, yes [23:57:21] that's actually what `include` does in Ruby [23:57:39] it imports methods from a given module into the current context/binding [23:58:12] modules also serve other functions like namespacing, etc. [23:59:01] in the most general terms, i'd say they are "containers for implementation"