[00:23:41] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ThrottleOverride: Deploy ThrottleOverride to beta cluster - https://phabricator.wikimedia.org/T182161#3814802 (10EddieGP) [01:28:02] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%) [03:40:16] Project mediawiki-core-code-coverage build #3175: 04FAILURE in 40 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3175/ [03:58:55] Yippee, build fixed! [03:58:55] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #599: 09FIXED in 2 min 54 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/599/ [04:51:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [10.0] [05:21:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [07:08:02] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [08:24:04] (03PS3) 10Hashar: docker: add dev tools to npm-test base images [integration/config] - 10https://gerrit.wikimedia.org/r/395555 [08:24:06] (03PS6) 10Hashar: I DONT KNOW WHAT I AM DOING [integration/config] - 10https://gerrit.wikimedia.org/r/395610 [08:24:20] 10commit-message-validator: Depends-On should come after Change-Id - https://phabricator.wikimedia.org/T182173#3815167 (10Tgr) [08:24:37] (03Abandoned) 10Hashar: docker: keep build-essential in npm images [integration/config] - 10https://gerrit.wikimedia.org/r/395648 (owner: 10Hashar) [08:29:05] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10Patch-For-Review, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3815194 (10Addshore) [08:40:25] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10Patch-For-Review, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3815235 (10Addshore) [08:54:06] addshore: good morning :) Congratulations on the Wikidata build step phase out ! [08:54:41] addshore: I could use mediawiki/extensions/DataTypes to be converted to a composer lib if at all possible [08:55:02] there is eg https://phabricator.wikimedia.org/T180172#3785736 which ask to run the QUnit tests with the MediaWiki test runner [08:55:09] but I guess it is an obsolete task now [09:28:54] hashar: *looks* [09:29:23] hashar: see https://phabricator.wikimedia.org/T180454 [09:29:51] amrked your task as invalid [09:40:33] addshore: awesome [09:40:39] :) [09:40:43] always glad to be of service [09:41:04] addshore: so what happens to DataTypes ? [09:41:10] is that included in another extension? :) [09:41:14] its being smushed into Wikibase [09:41:28] \o/ [09:42:28] (03CR) 10Addshore: [C: 031] Remove Wikidata extension [integration/config] - 10https://gerrit.wikimedia.org/r/395581 (https://phabricator.wikimedia.org/T173818) (owner: 10Ladsgroup) [09:42:57] hashar: ^^ can i just merge that and not worry about deploying it anywhere? [09:43:10] yup [09:43:16] (03CR) 10Addshore: [C: 032] Remove Wikidata extension [integration/config] - 10https://gerrit.wikimedia.org/r/395581 (https://phabricator.wikimedia.org/T173818) (owner: 10Ladsgroup) [09:43:18] addshore: well there might be other things to clean up [09:43:23] but we can catch up [09:43:25] yup, there are [09:43:29] don't worry, I'll do it :) [09:43:37] probably over the christmas period [09:43:41] eg: zuul/parameter_functions.py: 'WikidataPageBanner': ['Wikidata'], [09:43:53] Math apparently somehow depends on Wikidata bah [09:43:53] ooh yes indeed [09:44:02] lol, stupid math [09:44:06] i might actually have to look at that D: [09:44:16] well that is surely fixable by switching to Wikibase I guess [09:44:21] (03Merged) 10jenkins-bot: Remove Wikidata extension [integration/config] - 10https://gerrit.wikimedia.org/r/395581 (https://phabricator.wikimedia.org/T173818) (owner: 10Ladsgroup) [09:44:31] hashar: yes, should be switched ti Wikibase [09:44:38] though, it might need extra config who knows [09:44:48] it looks like it hooks into wikibase, so should be fine [09:46:17] (03PS1) 10Addshore: parameter_functions switch extensions to use Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/395704 [09:49:56] (03CR) 10Hashar: [C: 031] "Once deployed, please trigger a build for each of those extensions just to make sure they are still all fine? :)" [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (owner: 10Addshore) [09:51:45] (03PS1) 10Addshore: Load extensions of wikibase for wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/395705 [09:52:40] (03PS1) 10Addshore: Remove Wikidata.git from mirror-gerrit-repos script [integration/config] - 10https://gerrit.wikimedia.org/r/395707 [09:52:49] (03CR) 10jerkins-bot: [V: 04-1] Load extensions of wikibase for wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/395705 (owner: 10Addshore) [09:54:21] (03PS2) 10Addshore: Load extensions of wikibase for wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/395705 [09:54:31] (03PS2) 10Addshore: Remove Wikidata.git from mirror-gerrit-repos script [integration/config] - 10https://gerrit.wikimedia.org/r/395707 [09:55:09] hashar: I would really like to get https://gerrit.wikimedia.org/r/#/c/391564/ merged before christmas and also understand the new image process [09:55:17] so that I can work on CI over the christmas period :) [10:01:13] oooh, looks like it is documented, but needs some poking into lfie! https://www.mediawiki.org/wiki/Continuous_integration/Docker#Images_using_docker-pkg [10:03:54] !log docker push wmfreleng/npm:v2017.12.06.09.55 wmfreleng/npm-stretch:v2017.12.06.09.55 wmfreleng/npm-test:v2017.12.06.09.55 wmfreleng/npm-test-stretch:v2017.12.06.09.55 !!! wmfreleng/npm-browser-test:v2017.12.06.09.55 | https://gerrit.wikimedia.org/r/#/c/395555/ [10:03:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:04:33] (03PS4) 10Hashar: docker: add dev tools to npm-test base images [integration/config] - 10https://gerrit.wikimedia.org/r/395555 [10:09:57] (03PS5) 10Hashar: docker: add dev tools to npm-test base images [integration/config] - 10https://gerrit.wikimedia.org/r/395555 [10:10:44] (03CR) 10Hashar: [C: 032] "Jobs and containers pushed." [integration/config] - 10https://gerrit.wikimedia.org/r/395555 (owner: 10Hashar) [10:11:30] 10Scap, 10Services (watching): Scap fails to deploy in beta - https://phabricator.wikimedia.org/T182179#3815384 (10mobrovac) p:05Triage>03High [10:13:34] (03Merged) 10jenkins-bot: docker: add dev tools to npm-test base images [integration/config] - 10https://gerrit.wikimedia.org/r/395555 (owner: 10Hashar) [10:14:11] 10Scap, 10Services (watching): Scap fails to deploy in beta - https://phabricator.wikimedia.org/T182179#3815400 (10mobrovac) The offending patch seems to be {D907}. [10:15:22] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10Patch-For-Review, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3815405 (10Addshore) [10:16:45] 10Scap, 10Services (watching): Scap fails to deploy in beta - https://phabricator.wikimedia.org/T182179#3815406 (10mobrovac) In order to unblock myself, I manually fixed the offending line on `deployment-tin`: ``` --- deploy.py.bak 2017-12-06 10:15:04.972386035 +0000 +++ deploy.py 2017-12-06 10:15:26.26847387... [10:25:44] dcausse: your CirrusSearch backport to REL1_29 failed CI due to lack of a composer test command [10:26:02] dcausse: I have cherry picked the patch from master to REL1_29 and rebased your patch on top of it. Hopefully that will make it pass [10:26:12] hashar: thanks! [10:26:27] ;] [10:33:20] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3814704 (10zeljkofilipin) @mmodell I see that that task has #browser-tests tag. Let me know if yo... [10:33:52] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3815426 (10zeljkofilipin) [11:17:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [11:34:25] 10Release-Engineering-Team (Kanban), 10StructuredDiscussions, 10Browser-Tests, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), and 2 others: Flow: Migrate browser tests from Ruby to node.js - https://phabricator.wikimedia.org/T174591#3815595 (10zeljkofilipin) a:03zeljkofilipin [11:39:43] (03CR) 10Zfilipin: [C: 032] Do not run Ruby Selenium jobs for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/394111 (https://phabricator.wikimedia.org/T174591) (owner: 10Zfilipin) [11:40:58] (03Merged) 10jenkins-bot: Do not run Ruby Selenium jobs for Flow [integration/config] - 10https://gerrit.wikimedia.org/r/394111 (https://phabricator.wikimedia.org/T174591) (owner: 10Zfilipin) [11:43:32] 10Release-Engineering-Team (Kanban), 10StructuredDiscussions, 10Browser-Tests, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), and 2 others: Flow: Migrate browser tests from Ruby to node.js - https://phabricator.wikimedia.org/T174591#3815639 (10zeljkofilipin) 05Open>03Resolved Ruby tests and re... [11:47:40] 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3815650 (10zeljkofilipin) [12:18:04] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<100.00%) [12:38:05] RECOVERY - Free space - all mounts on integration-slave-jessie-1002 is OK: OK: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found) [13:31:37] (03PS1) 10KartikMistry: Add VE dependency for ContentTranslation [integration/config] - 10https://gerrit.wikimedia.org/r/395734 [15:12:45] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10Patch-For-Review, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818#3816193 (10Addshore) [15:15:02] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:24:02] (03PS2) 10Zfilipin: Update RuboCop Ruby gem [selenium] - 10https://gerrit.wikimedia.org/r/395513 (https://phabricator.wikimedia.org/T180878) [15:26:56] (03CR) 10Thiemo Mättig (WMDE): [C: 031] parameter_functions switch extensions to use Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (owner: 10Addshore) [15:27:17] (03CR) 10Thiemo Mättig (WMDE): [C: 031] Load extensions of wikibase for wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/395705 (owner: 10Addshore) [15:27:27] (03CR) 10Thiemo Mättig (WMDE): [C: 031] Remove Wikidata.git from mirror-gerrit-repos script [integration/config] - 10https://gerrit.wikimedia.org/r/395707 (owner: 10Addshore) [15:27:43] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:41:00] Project mediawiki-core-code-coverage build #3176: 04STILL FAILING in 40 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3176/ [15:47:41] addshore: Trying to @cover or @use not existing method "WikiPge::insertProtectNullRevision [16:04:20] greg-g: I wanted to show you this page today, in advance of tomorrow’s meeting…. https://wikitech.wikimedia.org/wiki/ORES/Deployment [16:37:53] awight: +1 cc no_justification ^ [16:38:43] cool. I’m around if you have any questions, I was hoping you could review our changes especially wrt. monitoring and emergency stuff. Didn’t want this to be a surprise tomorrow :) [16:39:25] For reference: T181010 [16:39:25] T181010: [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix - https://phabricator.wikimedia.org/T181010 [16:45:27] awight: So one thing I'm not seeing on that task, even though I've been a bit of a broken record about it....is why do we have to throw RuntimeExceptions on failure instead of failing back gracefully. [16:45:44] Because that's the one-sentence answer to "why does Ext:ORES cause 500s" [16:46:37] no_justification: That’s discussed and fixed in subtask T181191 [16:46:37] T181191: Make ORES-consuming pages more robust to ORES errors - https://phabricator.wikimedia.org/T181191 [16:47:18] Dur, my eyes skipped right past that one [16:48:20] I hid from sight by not saying “why are we crashing all over the hell” [16:48:41] !log Ran cleanupSpam.php on deploymentwiki [16:48:41] euphemistic disguise ftw [16:48:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:50:00] Hauskatze: nice nick [16:50:26] thanks Zppix [16:50:43] Np [16:51:33] awight: Sometimes I'm tempted to wrap all of index.php into a try...catch{} so nobody can throw exceptions at users :p [16:53:26] +1, I think I added the exceptions due to a few years lashed to the Java mast, where it’s normal to vomit NullPointerExceptions once per minute. (In a language explicitly designed to prevent null pointer exceptions.) [16:56:51] Hehhee [16:57:02] Yeah, exceptions have....gotten increasingly popular in MW & friends lately [16:57:20] Which sucks, because A) exceptions generally get throw back to users, which is less than ideal most of the time [16:57:34] And B) our current logging is less than stellar for them (I'd rather stuff logged as ERROR) [16:58:08] IMO we *should* have a top-level catch, I don’t think PHP has any nice facility to do that of course, so just a try…catch like you said. [17:03:49] Well, we catch, log them, then proceed to throw it back at the user as a generic error (which varnish then helpfully hides as a 503) [17:04:00] MWExceptionHandler & friends [17:04:04] (which is all a gigantic mess) [17:11:46] Ah right, it’s a styled fail at least :-). I wonder if there’s static code analysis to tell us that an exception can break out of its house? [17:12:52] I think exceptions are an elegant way to deal with some types of error passing, but yeah it’s key to keep them in your own court or be very explicit that the caller should be prepared to mop up the mess and degrade functionality. [17:14:16] The one that *really* bugs me has been InvalidArgumentException for USER PROVIDED INPUT [17:14:23] Of *course* users can give bad input [17:14:30] Returning a 503 to them is evil/mean [17:15:21] (03CR) 10Ladsgroup: [C: 031] Load extensions of wikibase for wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/395705 (owner: 10Addshore) [17:15:36] (03CR) 10Ladsgroup: [C: 031] parameter_functions switch extensions to use Wikibase [integration/config] - 10https://gerrit.wikimedia.org/r/395704 (owner: 10Addshore) [17:16:03] (03CR) 10Ladsgroup: [C: 031] "Yes please" [integration/config] - 10https://gerrit.wikimedia.org/r/395707 (owner: 10Addshore) [17:17:37] lolol [17:17:42] BAD USER [17:26:15] !log upgrading ELK on deployment-logstash2 - T178412 [17:26:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:26:21] T178412: Upgrade logstash cluster to elastic 5.5.x - https://phabricator.wikimedia.org/T178412 [17:36:35] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3816809 (10Halfak) [17:37:27] 10Release-Engineering-Team (Kanban), 10ORES, 10Operations, 10Scoring-platform-team, and 2 others: Git refusing to clone some ORES submodules - https://phabricator.wikimedia.org/T181552#3816812 (10Halfak) [17:38:14] (03CR) 10Hashar: [C: 032] dib: contint::hhvm is now a profile [integration/config] - 10https://gerrit.wikimedia.org/r/392926 (owner: 10Hashar) [17:39:29] (03Merged) 10jenkins-bot: dib: contint::hhvm is now a profile [integration/config] - 10https://gerrit.wikimedia.org/r/392926 (owner: 10Hashar) [17:39:53] (03CR) 10Hashar: [C: 032] "Nice one. And again congratulations for the wikidata.git phase out!" [integration/config] - 10https://gerrit.wikimedia.org/r/395707 (owner: 10Addshore) [17:40:32] 10Scap, 10Services (watching): Scap fails to deploy in beta - https://phabricator.wikimedia.org/T182179#3816831 (10mmodell) [17:41:09] (03CR) 10Hashar: [C: 04-1] Add VE dependency for ContentTranslation (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/395734 (owner: 10KartikMistry) [17:41:15] 10Scap, 10Services (watching): Scap fails to deploy in beta - https://phabricator.wikimedia.org/T182179#3816835 (10mmodell) @mobrovac: Sorry about that, patch incoming. [17:50:23] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:54:38] !log logstash upgrade on deployment-logstash2 completed, 5 minutes of logs lost during upgrade - T178412 [17:54:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:54:44] T178412: Upgrade logstash cluster to elastic 5.5.x - https://phabricator.wikimedia.org/T178412 [18:00:06] 10Release-Engineering-Team (Kanban), 10ORES, 10Operations, 10Scoring-platform-team, and 2 others: Git refusing to clone some ORES submodules - https://phabricator.wikimedia.org/T181552#3816966 (10mmodell) I'm not sure what to make of this one. I don't think T179013 ever effected production, so I'm not sure... [18:03:01] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3816990 (10mmodell) I'd like to push the latest scap code to production this week if I can get an opsen to upload the package. I'll create a... [18:05:16] (03PS6) 10Umherirrender: Changed settings for BlueSpice-repos [integration/config] - 10https://gerrit.wikimedia.org/r/394578 (owner: 10Robert Vogel) [18:05:28] (03CR) 10Umherirrender: [C: 031] Changed settings for BlueSpice-repos (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/394578 (owner: 10Robert Vogel) [18:35:38] 10Continuous-Integration-Config, 10MinusX: Add MinusX to MediaWiki extensions and PHP library repos - https://phabricator.wikimedia.org/T175794#3817121 (10Umherirrender) [18:35:59] paladox: Think we could get the ./tools/* directory copied over to its-phabricator so we can get standalone builds working? That and ./tools/eclipse/project.py :) [18:39:21] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:41:59] (03CR) 10Umherirrender: [C: 031] Archive the ActionEditSubmit extension [integration/config] - 10https://gerrit.wikimedia.org/r/394033 (https://phabricator.wikimedia.org/T180808) (owner: 10MarcoAurelio) [18:58:20] 10Gerrit, 10Release-Engineering-Team (Someday), 10Cleanup, 10Wikidata, and 3 others: Mark extension-Wikidata & wikidata-build-resources on Gerrit as ARCHIVED - https://phabricator.wikimedia.org/T181838#3803897 (10Umherirrender) - It is save to set Read-Only on gerrit and changed the description to begin wi... [19:06:05] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3817228 (10akosiaris) I can handle that. I 'll try and build it tomorrow and upload it if successful [19:23:30] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:53:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [19:54:31] (03CR) 10Hashar: [C: 032] Archive the ActionEditSubmit extension [integration/config] - 10https://gerrit.wikimedia.org/r/394033 (https://phabricator.wikimedia.org/T180808) (owner: 10MarcoAurelio) [19:55:32] (03Merged) 10jenkins-bot: Archive the ActionEditSubmit extension [integration/config] - 10https://gerrit.wikimedia.org/r/394033 (https://phabricator.wikimedia.org/T180808) (owner: 10MarcoAurelio) [19:56:19] (03CR) 10Hashar: "Deployed in Zuul" [integration/config] - 10https://gerrit.wikimedia.org/r/394111 (https://phabricator.wikimedia.org/T174591) (owner: 10Zfilipin) [20:01:27] 10Scap, 10Services (watching): Scap fails to deploy in beta - https://phabricator.wikimedia.org/T182179#3817755 (10mmodell) 05Open>03Resolved [20:05:33] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3817771 (10awight) [20:05:36] 10Release-Engineering-Team (Kanban), 10ORES, 10Operations, 10Scoring-platform-team, and 2 others: Git refusing to clone some ORES submodules - https://phabricator.wikimedia.org/T181552#3817772 (10awight) [20:05:58] 10Release-Engineering-Team (Kanban), 10ORES, 10Operations, 10Scoring-platform-team, and 2 others: Git refusing to clone some ORES submodules - https://phabricator.wikimedia.org/T181552#3793896 (10awight) [20:06:07] 10Release-Engineering-Team, 10ORES, 10Operations, 10Scoring-platform-team: Connection timeout from tin to new ores servers - https://phabricator.wikimedia.org/T181661#3797349 (10awight) [20:11:37] 10Release-Engineering-Team (Kanban), 10Phabricator, 10monitoring, 10Browser-Tests, 10User-zeljkofilipin: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160#3817815 (10mmodell) @zeljkofilipin: Thanks! Yeah I think Selenium might b... [20:13:30] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:34:29] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:36:25] 10Gerrit, 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, 10Operations, and 2 others: Plan migration of ORES repos to git-lfs - https://phabricator.wikimedia.org/T181678#3817844 (10demon) [20:36:31] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10Diffusion, and 5 others: Add gitlab to proxies/whitelist for mirroring to phabricator - https://phabricator.wikimedia.org/T181835#3817842 (10demon) 05Open>03Resolved a:03demon [20:36:33] (03PS7) 10Hashar: I DONT KNOW WHAT I AM DOING [integration/config] - 10https://gerrit.wikimedia.org/r/395610 [20:48:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [21:47:49] PROBLEM - Free space - all mounts on deployment-sca03 is CRITICAL: CRITICAL: deployment-prep.deployment-sca03.diskspace._srv.byte_percentfree (<30.00%) [21:51:23] (03PS8) 10Hashar: I DONT KNOW WHAT I AM DOING [integration/config] - 10https://gerrit.wikimedia.org/r/395610 [21:53:03] (03CR) 10Hashar: "That one works for me locally with some other nasty trick." [integration/config] - 10https://gerrit.wikimedia.org/r/395610 (owner: 10Hashar) [22:02:02] 10Scap, 10ORES, 10Scoring-platform-team: ORES virtualenv deployment step fails intermittently - https://phabricator.wikimedia.org/T182258#3818204 (10awight) [22:23:10] (03PS9) 10Hashar: I DONT KNOW WHAT I AM DOING [integration/config] - 10https://gerrit.wikimedia.org/r/395610 [22:39:04] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [23:30:46] composer breaks CI: https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/25179/console [23:30:56] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:36:10] PROBLEM - Puppet errors on deployment-netbox is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:25] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:40:07] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:41:54] greg-g: what's the appropriate phab tag for strange errors breaking CI? [23:42:18] T182266, specifically [23:42:19] T182266: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266 [23:44:02] 10Gerrit, 10Release-Engineering-Team (Someday), 10Cleanup, 10Wikidata, and 3 others: Mark extension-Wikidata & wikidata-build-resources on Gerrit as ARCHIVED - https://phabricator.wikimedia.org/T181838#3818535 (10Addshore) If we need sub tickets for each of these then they should be made under the main kil... [23:44:14] tgr: ci-config [23:46:37] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818539 (10Tgr) [23:50:19] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818543 (10Tgr) [[https://github.com/composer/composer/issues/2198|composer#2198]] suggests this could be caused by an outdated Composer executable. [23:55:47] 10Continuous-Integration-Config, 10Composer: Composer\Downloader\TransportException in mwext-testextension-hhvm-jessie - https://phabricator.wikimedia.org/T182266#3818553 (10Tgr) Per https://gerrit.wikimedia.org/r/#/q/projects:mediawiki/ seems to have started half an hour ago.