[00:58:37] eddiegp: I've never seen it be that slow, it does sound like it might be related to nfs. Personally I've had good experience using sshfs for the same purpose - it is fairly efficient for a user mode filesystem [00:59:03] (sshfs is also easy to set up) [01:26:07] Project beta-scap-eqiad build #145157: 04FAILURE in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/145157/ [01:36:17] Project beta-scap-eqiad build #145158: 04STILL FAILING in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/145158/ [01:46:11] Project beta-scap-eqiad build #145159: 04STILL FAILING in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/145159/ [01:56:43] Yippee, build fixed! [01:56:44] Project beta-scap-eqiad build #145160: 09FIXED in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/145160/ [04:07:44] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #321: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/321/ [06:58:35] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [07:03:09] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:08:34] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [09:10:52] 10Beta-Cluster-Infrastructure: Special:Version displays incorrect information for what commit is deployed there - https://phabricator.wikimedia.org/T159520#3074603 (10Nikerabbit) Now it shows ``` ContentTranslation – (27ce92a) 21:26, 3 March 2017 ``` But https://gerrit.wikimedia.org/r/#/c/340923/ has been merge... [10:29:53] (03PS3) 10Zfilipin: Upgrade to selenium-webdriver 3.2 [selenium] - 10https://gerrit.wikimedia.org/r/340317 (https://phabricator.wikimedia.org/T158074) [10:52:09] 10Gerrit, 06Release-Engineering-Team: Update gerrit to 2.13.6 - https://phabricator.wikimedia.org/T158946#3075013 (10Paladox) @demon scratch that it seems the ui is broken now, see https://gerrit-review.googlesource.com/#/c/96370/ (bottom). So we doint want to enable that config we should continue using ldap... [10:53:38] 10Beta-Cluster-Infrastructure: Special:Version displays incorrect information for what commit is deployed there - https://phabricator.wikimedia.org/T159520#3070282 (10hashar) That must be MediaWiki GitInfo being cached for X time. I have no idea how it is cached though or whether there is a setting to tweak the... [10:58:42] (03CR) 10Hashar: [C: 032] Upgrade to selenium-webdriver 3.2 [selenium] - 10https://gerrit.wikimedia.org/r/340317 (https://phabricator.wikimedia.org/T158074) (owner: 10Zfilipin) [11:00:28] (03Merged) 10jenkins-bot: Upgrade to selenium-webdriver 3.2 [selenium] - 10https://gerrit.wikimedia.org/r/340317 (https://phabricator.wikimedia.org/T158074) (owner: 10Zfilipin) [11:01:13] (03CR) 10jenkins-bot: Upgrade to selenium-webdriver 3.2 [selenium] - 10https://gerrit.wikimedia.org/r/340317 (https://phabricator.wikimedia.org/T158074) (owner: 10Zfilipin) [12:22:20] (03PS1) 10Addshore: Add extension-phan-generic to Wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/341304 [13:49:08] PROBLEM - Puppet run on buildlog is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:38:45] hashar: jenkins when issues a main test due to a commit change and then recieving a cr+2 before its done with the test will cause the test to v:-1 and then it sucessfully merge with v+2 [14:41:29] hashar: see https://gerrit.wikimedia.org/r/#/c/340983/ for what i am talking about [14:44:27] Zppix: mediawiki-extensions-qunit-jessie FAILURE in 4m 28s [14:44:32] that job failed for whatever reason [14:44:57] hashar: it probably failed due to the gate and submit being ran at the same ime [14:44:58] time [14:45:04] nop [14:45:18] each builds run on separate instances that are all the same [14:46:04] maybe the instance the test ran on has a bad build/ [14:46:06] 00:04:23.435 Chrome 55.0.2883 (Linux 0.0.0) ve.ce.Surface (MW) handleLinearDelete FAILED [14:46:06] 00:04:23.435 afterEach failed on handleLinearDelete: Pending AJAX requests: 0 (active: 1) [14:46:13] Error: Pending AJAX requests: 0 (active: 1) [14:46:17] some request took to long I guess [14:46:36] it was an edit to the credits file a text doc ajax wouldnt have a reason to fail... [14:46:44] yeah [14:46:50] that is an issue in the test suite itself [14:46:58] or maybe a cosmic ray caused it to fail [14:47:10] hashar: maybe it should be rebuilt? [14:47:12] some tests randomly fail [14:47:21] well the job passed on CR+2 [14:47:29] but it was ran on a different instance [14:47:34] so dont bother until it happens more often [14:47:57] yeah that is the point. For most jobs, each build is made on a fresh instance [14:48:05] and the instance is destroyed upon build completion [14:48:46] i may do some testing on test gerrit to see if i cant make it do it again, is jenkins set up on the test gerrit? [15:02:06] Project mediawiki-core-code-coverage build #2616: 15ABORTED in 2 min 5 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2616/ [15:13:45] 06Release-Engineering-Team, 07Performance, 10Phabricator (Search): Extremely slow Phabricator search (1m 45s) - https://phabricator.wikimedia.org/T159575#3075899 (10mmodell) Ok this was caused by a mistake in code I wrote which was causing us to inadvertently query the mysql fulltext index. It's now hotfixe... [15:13:59] 06Release-Engineering-Team, 07Performance, 10Phabricator (Search): Extremely slow Phabricator search (1m 45s) - https://phabricator.wikimedia.org/T159575#3075901 (10mmodell) p:05High>03Normal a:03mmodell [15:14:25] 10Browser-Tests-Infrastructure, 10MediaWiki-General-or-Unknown, 07JavaScript, 13Patch-For-Review, and 2 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3075904 (10zeljkofilipin) [15:14:50] 10Beta-Cluster-Infrastructure: Special:Version displays incorrect information for what commit is deployed there - https://phabricator.wikimedia.org/T159520#3075905 (10Nikerabbit) They are cached for 24 hours using keys of format `specialversion-ext-version-text-NAME-COREID` with no way to override. [15:15:29] So I have a question, and I am vaguely aware it may severely slow my deployment process. We have Extension:3d almost reviewed and ready to go to the beta cluster. But we also have https://phabricator.wikimedia.org/source/3d2png/ which is a separate codebase, and may need to be deployed via puppet... [15:16:05] My thinking is that this may need to be packaged for Debian, but I don't know fully what that looks like, so I'm hoping someone here has the insight and time to help me out getting it done. [15:22:35] marktraceur: isnt that part of thumbor ? [15:23:07] an option would be to bundle 3d2png direclty in the 3d extension [15:23:17] similar to oojs which is flattened in mediawiki/core [15:23:30] I mean, it could be part of thumbor, but not currently [15:23:59] a .deb package, I have no idea how painful it is going to be given you will have to package the dependencies [15:24:06] We could package 'er up for the 3d extension, but that might be a pain...there are some NPM dependencies at least, maybe some Debian dependencies at worst [15:24:27] and the last option is to use a deploy repo that ships 3d2png + the node_modules dir and deploy that with scap [15:25:00] deploy.git repo + scap is how deployment is handled for the various mediawiki/services/ [15:25:05] so maybe you want to go that way [15:25:50] then if you have no binary modules, maybe it is easier to ship a snapshot of 3d2png+node_modules directly in the 3d mw extension [15:26:43] Well, just looked at the Vagrant module, looks like it requires some stuff from the deb repos. [15:27:01] Some of it might be included from other thumbnailing stuff, but I dunno. [15:27:14] * marktraceur wonders if gilles would comment briefly [15:27:36] well for the .deb [15:27:44] I guess you "just" need a puppet class that list them out [15:27:59] That would work fine, yeah [15:28:14] and include it in whatever puppet class we already have applied on the thumb generators [15:28:23] or whatever class of servers 3d2png will act on :] [15:28:51] That's also not super clear to me, but I'll try to work it all out... [15:29:20] modules/mediawiki/manifests/packages/ has a few manifests [15:29:24] It would make perfect sense to put it on the thumbnailing servers, but I'm not 100% sure how to configure the extension for that [15:29:26] like multimedia.pp [15:29:43] maybe you will be fine by just adding the deps there [15:31:55] I'll give it a shot [15:36:36] Should I be discussing this with opsen first? [15:41:51] marktraceur: well you can bring up all your question on the ops mailling list [15:42:00] that reach out to both ops and releng (among others) [15:42:04] Ugh, I hate making noise on that list [15:42:07] But okay. [15:42:16] it is not noise!!! :) [15:42:30] I'm going to wait for gilles to respond, but if he's not here for a bit I'll email ops [15:42:39] you wanna touch the infrastructure, and ops is teh best place for that I guess [15:42:48] hashar: I mean, I get internal team stuff from ops@, which is annoying, so I'm sensitive about it [15:42:57] and you can fill in a task with the requirement / explanation of what you want [15:43:13] I believe the "ops" team has its own private list [15:43:17] I'm still not entirely sure what needs to be done, so that's necessarily part of the question [15:43:23] yeah [15:43:24] well [15:43:34] you want the mediawiki 3d extension to be deployed [15:43:38] it generate some whatever model [15:43:46] Like I'm happy to dump 3d2png's dependencies into its repo and deploy that, but it's a whole other thing [15:43:48] which has to be viewable by end users [15:44:06] Yeah, I'll start there. [15:44:22] with 3d2png being a renderer for that whatever model to png [15:44:32] STL :) [15:44:33] (which might or might not be the best solution but I digress) [15:44:49] It's the solution we've got right now [15:45:07] And we're only going to beta, so worst case we don't move to production while we look for a better pipeline [15:45:15] yup :] [16:03:37] !log Jenkins upgrading "Git client plugin" 1.19.6 to 2.3.0 [16:03:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:33:25] 06Release-Engineering-Team, 10Scap: scap should handle changes to .gitmodules - https://phabricator.wikimedia.org/T157694#3076151 (10thcipriani) 05Open>03Resolved a:03Halfak @Halfak thank you for the fix! This is in the currently deployed version of scap (went out with version 3.5.2-1). [16:43:29] 06Release-Engineering-Team, 10Scap: scap should handle changes to .gitmodules - https://phabricator.wikimedia.org/T157694#3076215 (10Halfak) \o/ [16:52:53] (03PS1) 10Hashar: Clean JJB configuration [integration/config] - 10https://gerrit.wikimedia.org/r/341347 [16:54:41] (03CR) 10Hashar: [C: 032] "Noop in JJB" [integration/config] - 10https://gerrit.wikimedia.org/r/341347 (owner: 10Hashar) [16:55:41] (03Merged) 10jenkins-bot: Clean JJB configuration [integration/config] - 10https://gerrit.wikimedia.org/r/341347 (owner: 10Hashar) [17:19:31] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 03Language-2017 Sprint 3, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3076267 (10hashar) So one has to clone and wfLoadExtensions: ``` mediawiki/extensions/Conten... [17:22:40] 10MediaWiki-Releasing, 10MediaWiki-Containers, 06Services, 15User-mobrovac: Ready-to-use Docker package for MediaWiki - https://phabricator.wikimedia.org/T92826#1121279 (10DanBolser) Hello, What is the status of this project? [17:33:56] 10Gerrit, 07Regression, 07Upstream: Gerrit doesn't display committed images - https://phabricator.wikimedia.org/T153085#3076300 (10Paladox) Polygerrit will be exposed in gerrit 2.14. You will be able to view images in diffs again in polygerrit :) [17:35:09] 10Gerrit, 07Regression, 07Upstream: Gerrit doesn't display committed images - https://phabricator.wikimedia.org/T153085#3076304 (10Niedzielski) {icon thumbs-up}! [17:46:35] * addshore pokes hasharErrand to poke him when he gets back! [17:55:33] * addshore needs to work out how to correctly create a database on beta ;) [18:00:13] 10Gerrit, 07Regression, 07Upstream: Gerrit doesn't display committed images - https://phabricator.wikimedia.org/T153085#2869243 (10demon) >>! In T153085#3076300, @Paladox wrote: > Polygerrit will be exposed in gerrit 2.14. You will be able to view images in diffs again in polygerrit :) That's not a solution... [18:01:14] 10Gerrit, 07Regression, 07Upstream: Gerrit doesn't display committed images - https://phabricator.wikimedia.org/T153085#3076360 (10Paladox) Yes but upstream are not accepting any usability improvements for gwt, though you can submit a patch to see though. [18:01:26] RainbowSprinkles hi, i got polygerrit finally working on /r/ [18:01:39] urls [18:01:40] with this https://gerrit-review.googlesource.com/#/c/99004/ patch [18:01:48] working demo at http://gerrit-new.wmflabs.org/r/q/status:open [18:02:21] 10Gerrit, 07Regression, 07Upstream: Gerrit doesn't display committed images - https://phabricator.wikimedia.org/T153085#3076361 (10demon) >>! In T153085#3076360, @Paladox wrote: > Yes but upstream are not accepting any usability improvements for gwt, though you can submit a patch to see though. That's just... [18:02:52] http://gerrit-new.wmflabs.org/r/q/status:open?polygerrit=1 it's not perfect but it at least now gets most of the features of polygerrit working :) [18:16:51] RainbowSprinkles they just switched polygerrit for default ui on https://gerrit-review.googlesource.com/ even though i put two blockers, no internet explorer support and ios 10.3 seems broken with polygerrit. [18:17:21] there excuse is it is a website for developing gerrit so users should know how to change back to the old ui [18:17:22] lol [18:17:54] I mean, if they wanna shoot themselves in the foot with a broken UI on their website, that's their prerogative ;) [18:18:02] But I am not forcing it on my users :) [18:18:54] :) [18:38:50] RainbowSprinkles, i forgot did we cherry pick this commit https://gerrit-review.googlesource.com/#/c/92610/ ? [18:39:14] Since it seems that there is no commit that cherry picked it so i did it here https://gerrit-review.googlesource.com/#/c/99290/ [18:42:04] RainbowSprinkles! [18:42:21] any idea how i should correctly go about creating a db on the deployment-pre / beta db server? [18:42:35] or is the correct way just log in as root and jab around and make it? ;) [18:42:40] (probably) [18:45:41] 10Deployment-Systems, 06Release-Engineering-Team, 06Operations: tin.eqiad.wmnet / partition is full - https://phabricator.wikimedia.org/T158358#3076653 (10Ottomata) 05Open>03Resolved a:03Ottomata / is no longer full, and there are other tickets to resolve the bigger problems. Resolving. [18:46:44] addshore: To what end? A new wiki? [18:46:58] nope, a database for an extension [18:47:19] a db shared between multiple wikis [18:52:12] 10Gerrit, 06Release-Engineering-Team, 06Operations: Decide weather to disable drafts in gerrit - https://phabricator.wikimedia.org/T158656#3076685 (10Ottomata) p:05Triage>03Low a:03demon Triaging, feel free to re-assign as needed. [18:58:38] aaaah mwscript --wiki=aawiki sql.php [18:58:41] i guess [19:03:22] aawiki serves for a purpose finally [19:07:27] addshore: bonsoir. I was unable to follow up on your friday email :( [19:07:45] hashar: Thats okay! I think I have just about figured everything out :D [19:07:55] Including the skiing and snowboarding on snow ;) [19:08:11] Just about to push Cognate to beta and see if it works :D [19:13:44] 06Release-Engineering-Team, 10MediaWiki-API, 10Monitoring, 10Parsoid, and 2 others: API action=parsoid-batch not available on Graphite - https://phabricator.wikimedia.org/T152776#3076767 (10Ottomata) I don't think there are any ops action items here, so I'm removing the tag. [19:15:24] RainbowSprinkles lol upstream gerrit issue tracker has blocked the word phabricator [19:19:44] 10Continuous-Integration-Config, 06Operations, 07Puppet: Get rid of "import realm.pp" in manifests/site.pp - https://phabricator.wikimedia.org/T154915#3076815 (10Ottomata) p:05Triage>03Normal [19:29:31] paladox: That's not a lol, it sounds like you've annoyed them... [19:29:51] But i didnt paste that many phabricator links though [19:30:33] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Operations: Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org - https://phabricator.wikimedia.org/T157038#3076843 (10Ottomata) p:05Triage>03Normal [19:34:25] RainbowSprinkles the email bug should be fixed in https://gerrit-review.googlesource.com/#/c/99178/ now :) [19:34:59] So now users imboxes should not be full of gerrit emails. Since if someone got cced then it sent an email to the original user too. [19:35:55] ah [19:36:07] RainbowSprinkles it just seems to be polygerrit blocking the word phabricator. [19:36:13] reports against gerrit works [19:36:35] Polygerrit at one stage made all issues reported against polygerrit hidden. [19:37:35] 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Wikipedia-Android-App-Backlog, 07Spike, 07Technical-Debt: Investigate how to improve Android CI performance and stability - https://phabricator.wikimedia.org/T158014#3076915 (10Ottomata) p:05Triage>03Normal I don't think ops has specific... [19:47:31] 10Gerrit, 06Release-Engineering-Team, 06Operations: Decide weather to disable drafts in gerrit - https://phabricator.wikimedia.org/T158656#3077021 (10demon) 05Open>03Resolved Sorry, this was never an #operations issue. Question was decide whether to disable or not? I decided not to. Really, there's noth... [19:47:52] 06Release-Engineering-Team, 10Phabricator, 07Technical-Debt: Replace deprecated phabricator conduit api calls in phab_epipe.py file - https://phabricator.wikimedia.org/T159043#3077024 (10Ottomata) [19:48:03] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: can loose data if it crashes - https://phabricator.wikimedia.org/T159743#3077029 (10Paladox) [19:48:36] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: can loose data if it crashes - https://phabricator.wikimedia.org/T159743#3077045 (10Paladox) [19:48:38] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3077044 (10Paladox) [19:51:46] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: can loose data if it crashes - https://phabricator.wikimedia.org/T159743#3077059 (10Ottomata) p:05Triage>03Normal [19:52:37] 10Gerrit, 06Release-Engineering-Team: Gerrit: inline edit references are not updated - https://phabricator.wikimedia.org/T159744#3077060 (10Paladox) [19:55:00] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3077088 (10Paladox) [20:01:48] 10Gerrit, 06Release-Engineering-Team: Gerrit: inline edit references are not updated - https://phabricator.wikimedia.org/T159744#3077115 (10demon) Was anyone actually noticing or complaining about this locally? If not there's zero reason to file a local bug. [20:15:49] Project beta-update-databases-eqiad build #15467: 04FAILURE in 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/15467/ [20:16:46] ^^ thats my fault [20:16:52] looking into it now [20:20:28] Project beta-update-databases-eqiad build #15468: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/15468/ [20:21:09] well, thats clearly also me :D [20:28:22] and that job should start working again now, per https://gerrit.wikimedia.org/r/#/c/341380/ [20:36:08] 10Gerrit, 06Release-Engineering-Team: Gerrit: inline edit references are not updated - https://phabricator.wikimedia.org/T159744#3077288 (10Dzahn) p:05Triage>03Low [20:37:02] 10Gerrit, 06Release-Engineering-Team: Gerrit: inline edit references are not updated - https://phabricator.wikimedia.org/T159744#3077060 (10Dzahn) What demon said, i don't think we had local complaints and we'd just be duplicating the upstream ticket, afaict. [20:40:28] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 0 below the confidence bounds [20:41:35] thcipriani: hashar unsure if ^ is an actual issue [20:42:13] chasemp: it is not. I thought I had acked that service for a month or so [20:42:18] guess I should just remove it [21:02:29] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is OK: OK: No anomaly detected [21:03:59] chasemp: thcipriani: https://gerrit.wikimedia.org/r/341402 Change Zuul Gearman alarm to a simple threshold :] [21:04:08] will loop back with daniel [21:15:52] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3077508 (10Paladox) [21:15:55] 10Gerrit, 06Release-Engineering-Team: Gerrit: inline edit references are not updated - https://phabricator.wikimedia.org/T159744#3077506 (10Paladox) 05Open>03declined Yep, sorry. [21:17:54] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 03Language-2017 Sprint 3, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3077516 (10hashar) a:03hashar I reproduced it. The TLDR is that the tests fail on Chromium... [21:21:31] Yippee, build fixed! [21:21:31] Project beta-update-databases-eqiad build #15469: 09FIXED in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/15469/ [21:28:10] 10Continuous-Integration-Config: Castor: mediawiki-core-qunit-jessie node_modules cache ineffective - https://phabricator.wikimedia.org/T159591#3077578 (10Krinkle) 05Open>03declined On IRC, @hashar reminded me that we aren't using castor to cache the `$WORKSPACE/node_modules` directory. Instead, we use casto... [21:29:20] hashar: karma-mocha-reporter looks cool. The project also looks fairly healthy and well-maintained. [21:29:28] I'd +2 switching core. [21:29:47] Krinkle: yeah I found that gem while trying to figure out a qunit failure for an extension [21:29:56] will polish the dirty patch I made [21:30:12] hashar: the current reporter only shows the name of the failing test, not the test that ran before it which can sometimes be usefl. [21:30:21] the current report is siimilar to PHPUnit. [21:30:28] But yeah, this one is nicer. [21:30:35] and only 1 dependency [21:33:37] Krinkle: https://gerrit.wikimedia.org/r/341377 build: karma now reports with mocha formatter :] [21:33:49] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:34:02] before: https://integration.wikimedia.org/ci/job/mwext-qunit-jessie/8842/console [21:34:07] after https://integration.wikimedia.org/ci/job/mwext-qunit-jessie/8844/console [21:34:32] this way hopefully we get the error next to the test [21:34:34] hashar: And Karma QUnit section in jenkins still works [21:34:46] oh I havent checked tha [21:34:46] it broke a few weeks ago, but fixed it :) [21:34:51] great!! [21:35:37] hashar: hm,, that before link looks like a custom build, not a normal one [21:35:38] debug mode? [21:35:49] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:40:49] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [21:40:49] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [21:40:52] Krinkle: yeah probbly [21:40:57] I think I added &debug somehow [21:42:00] the good thing with mocha report, is that it makes error stand out a bit nicely [21:49:44] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 03Language-2017 Sprint 3, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3077662 (10hashar) And of course the Nodepool instances have a stall apt configuration for s... [21:53:28] PROBLEM - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 3 below the confidence bounds [21:56:39] ACKNOWLEDGEMENT - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 3 below the confidence bounds amusso It is broken and being replaced. [21:56:43] \o/ [21:57:24] RECOVERY - Work requests waiting in Zuul Gearman server https://grafana.wikimedia.org/dashboard/db/zuul-gearman on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] [21:57:37] 06Release-Engineering-Team, 10Page-Previews, 06Reading-Web-Backlog: Generate compiled assets from continuous integration - https://phabricator.wikimedia.org/T158980#3077692 (10Jdlrobson) @hashar and @zeljkofilipin do you have any bandwidth over the next 2 weeks to help us with this? [21:58:56] hashar: I think you missed a PM from me? :o [21:59:19] yeah that is unsurprising [21:59:25] my IRC Clients typically dont ping me on PM [21:59:45] I mostly have the problem that I don't see PMs because I'm in too many chans [21:59:56] 65 currently [22:00:03] so yeah you could ask in this channel :] [22:00:23] and I cant update the topic [22:00:34] I am not even sure i have operator channel here (maybe I do) [22:00:40] hashar: op yourself before: /msg ChanServ OP #wikimedia-releng [22:00:45] you have, IIRC [22:01:02] hashar: fwiw creation times have been normal all day https://graphite.wikimedia.org/render/?width=973&height=488&_salt=1488408597.388&lineMode=connected&target=servers.labnet1001.nova.verify.creation&target=servers.labnet1001.nova.verify.fullstack&lineWidth=3&title=Fullstack%20test%20and%20Creation%20times&fontBold=true&bgcolor=FFFFFF&graphOnly=false&hideAxes=false&from=-8h [22:01:05] -ChanServ- 3 Hashar +Viortv (team-member) [modified 2y 7w 4d ago] [22:01:09] that's enough ;) [22:01:53] hashar: There you go ^ :) [22:01:56] Now you can /topic [22:02:15] you are so kind :] [22:02:58] Sagan: so that link https://bit.ly/relirclog [22:03:07] yeah [22:03:07] who has the power to change it ? :D [22:03:17] * Sagan has :o [22:03:23] ;D [22:03:24] hashar: Anyone who's op'd [22:03:28] Channel has +t [22:03:44] RainbowSprinkles: I guess he meant the shortened link ;) [22:03:58] Oh, nvm [22:04:01] * RainbowSprinkles wanders to find food [22:04:37] chasemp: yup I noticed that and nodepool has the instances in less than a minute now \o/ [22:05:13] chasemp: and feel free to hijack the lame board I made on https://grafana.wikimedia.org/dashboard/db/labs-nova-fullstack [22:05:14] we are unsure why those labvirts are effected atm but it seems clear they are the culprite [22:07:43] hashar: yeah it's pretty decent, I was going to explore making those dots connected so it's not so scatter plot [22:08:48] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: can loose data if it crashes - https://phabricator.wikimedia.org/T159743#3077029 (10greg) I don't see why we need this task. There's always tons of changes that happen in an upstream that fix minor along with major issues. We don't need a task for t... [22:08:59] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: can lose data if it crashes - https://phabricator.wikimedia.org/T159743#3077716 (10greg) [22:10:01] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3077728 (10greg) [22:10:03] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: can lose data if it crashes - https://phabricator.wikimedia.org/T159743#3077029 (10greg) 05Open>03Invalid Per T159744 also invaliding. [22:10:06] 10Gerrit, 06Release-Engineering-Team: Gerrit: inline edit references are not updated - https://phabricator.wikimedia.org/T159744#3077060 (10greg) 05declined>03Invalid [22:10:09] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#2964656 (10greg) [22:12:57] greg-g: lol :o [22:13:24] interesting hack [22:19:40] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 03Language-2017 Sprint 3, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3077819 (10hashar) I dont quite know what is broken in the apt repository. I am regeneratin... [22:45:01] (03PS1) 10Hashar: dib: prevent apt from using a web proxy [integration/config] - 10https://gerrit.wikimedia.org/r/341455 [22:47:02] (03PS2) 10Hashar: dib: prevent apt from using a web proxy [integration/config] - 10https://gerrit.wikimedia.org/r/341455 [23:00:58] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 03Language-2017 Sprint 3, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#3077992 (10hashar) The generation of the Jessie image got broken end of December. https://ge... [23:02:57] 10Continuous-Integration-Config, 10ContentTranslation, 03Language-2017 Sprint 2, 03Language-2017 Sprint 3, and 3 others: mwext-qunit-jessie test fails on unrelated change - https://phabricator.wikimedia.org/T153038#2867222 (10Zppix) Unrelated but it appears this happens on mediawiki's own version of this job [23:22:07] (03PS1) 10Krinkle: performance-webpagetest: Raise runs from 5 to 6 for desktop tests [integration/config] - 10https://gerrit.wikimedia.org/r/341461 [23:56:01] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T157639#3078218 (10demon) 05Open>03Resolved [23:56:37] 06Release-Engineering-Team, 10Scap: Support shallow clones - https://phabricator.wikimedia.org/T157149#3078222 (10demon) a:05demon>03None Unlicking cookie in case I don't get to this ASAP