[00:02:13] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:28] PROBLEM - Puppet staleness on deployment-salt02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [02:37:28] RECOVERY - Puppet staleness on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [3600.0] [03:32:54] PROBLEM - Puppet run on deployment-db03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:12:54] RECOVERY - Puppet run on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [04:54:34] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:34:32] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:46:05] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:21:04] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [06:44:28] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:24:28] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:53:25] (03PS2) 10Niedzielski: Limit performance jobs IRC notification channels [integration/config] - 10https://gerrit.wikimedia.org/r/308579 [08:00:00] (03CR) 10Hashar: [C: 04-1] "The ircbot setting "matrix-notifier" is for multi configuration jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/308579 (owner: 10Niedzielski) [08:04:44] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [08:16:55] 10Continuous-Integration-Infrastructure, 06Labs, 13Patch-For-Review, 07Wikimedia-Incident: Nodepool instance instance creation quota management - https://phabricator.wikimedia.org/T143016#2610325 (10hashar) The https://horizon.wikimedia.org/project/instances/ shows proper values / instances now. [08:44:43] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:59:28] !sal [08:59:28] https://tools.wmflabs.org/sal/releng [09:00:26] !log Trying to refresh Nodepool Jessie image . Image properties have been dropped, should fix it [09:00:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:00:50] hashar: any idea why this is not merged? o.O https://gerrit.wikimedia.org/r/#/c/308563/ [09:01:02] hasn't been reviewed ? [09:01:04] lacked a CR+2 [09:01:05] I don't see anything in zuul https://integration.wikimedia.org/zuul/ [09:01:17] in Gerrit depends on a change that is still open? [09:01:20] hashar: it does have a +2 [09:01:27] hm, good idea, checking [09:01:39] look at the change status in Gerrit [09:01:43] top left: [09:01:49] Change 12345 - [09:01:54] no, looks like it depends on nothing [09:01:59] if you get something like "merge pending" that is Gerrit being bus [09:02:03] Change 308563 - Ready to Submit [09:02:11] yeah cause it has both CR+2 and V+2 [09:02:13] but, we don't have to submit manually, right? [09:02:24] oh, v+2 messed things up? [09:02:28] then is Zuul even configured for that repo? [09:02:43] Tobi_WMDE_SW_NA ^ [09:02:47] oh [09:02:53] that is change to drop all materials [09:02:55] just submit it [09:02:58] hashar: not sure, probably [09:03:04] ok, will do [09:03:16] :] [09:03:31] hm, but how? [09:03:40] there used to be submit button, as far as I remember [09:03:41] zeljkof: thx. can you submit it? I don't see a submit button [09:03:44] but can not see it [09:03:47] oh [09:03:49] maybe you are not allowed to submit [09:03:54] :D [09:03:55] i will :] [09:04:03] hashar: please, and thank you [09:04:04] I got the button showing up for some reason [09:04:08] !solved [09:04:12] thanks! [09:04:16] other trick is [09:04:20] looks like both me and Tobi_WMDE_SW could not submit [09:04:21] one can "git push" [09:04:32] hashar: from local machine? [09:04:34] and Gerrit should close the change and comment "$USER has pushed commit" [09:04:35] yeah [09:04:36] hashar: thx! ok, will try to remember for next time.. :) [09:04:42] and when pushing [09:04:44] I would hesitate to push :) [09:04:50] you get a bunch on your local side [09:04:59] that shows the progress of changes being analysed and closed for you [09:05:31] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2610426 (10Tobi_WMDE_SW) [09:06:35] !log nodepool.SnapshotImageUpdater: Image ci-jessie-wikimedia-1473152393 in wmflabs-eqiad is ready [09:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:08:49] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2610433 (10Tobi_WMDE_SW) @Paladox I've set https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/browsertests to read-only now. Is... [09:11:48] !log nodepool.SnapshotImageUpdater: Image ci-trusty-wikimedia-1473152801 in wmflabs-eqiad is ready [09:11:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:32:56] (03CR) 10Legoktm: "The test failure is due to ." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308162 (owner: 10Paladox) [09:45:33] (03PS3) 10Legoktm: Update squizlabs/php_codesniffer to 2.7.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308162 (owner: 10Paladox) [09:45:53] (03CR) 10Legoktm: [C: 032] Update squizlabs/php_codesniffer to 2.7.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308162 (owner: 10Paladox) [09:46:40] (03Merged) 10jenkins-bot: Update squizlabs/php_codesniffer to 2.7.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308162 (owner: 10Paladox) [09:56:46] (03PS1) 10Legoktm: Add test to verify SpaceBeforeClassBraceSniff handles extends [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308700 (https://phabricator.wikimedia.org/T142343) [09:57:13] 10MediaWiki-Codesniffer, 13Patch-For-Review: SpaceBeforeClassBraceSniff should also handle "class Foo extends Bar{" - https://phabricator.wikimedia.org/T142343#2610492 (10Legoktm) As far as I can tell, this was always supported. [09:57:21] 10MediaWiki-Codesniffer, 13Patch-For-Review: SpaceBeforeClassBraceSniff should also handle "class Foo extends Bar{" - https://phabricator.wikimedia.org/T142343#2610493 (10Legoktm) 05Open>03Invalid [10:03:52] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:08:44] (03CR) 10Paladox: "Thankyou" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308162 (owner: 10Paladox) [10:12:15] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure, 13Patch-For-Review: Nodepool image metadata show: true is invalid/rejected by OpenStack API: {u'show': u'true'} is not of type 'string' - https://phabricator.wikimedia.org/T144753#2610543 (10hashar) 05Open>03Resolved The issue has... [10:15:56] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Operations, 10Ops-Access-Requests, and 3 others: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270#2458739 (10MoritzMuehlenhoff) I don't think access to the security IRC channel is a go... [10:19:57] 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Operations, 06Security-Team, and 2 others: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270#2610549 (10MoritzMuehlenhoff) [10:20:37] (03PS1) 10Legoktm: [WIP] Use upstream sniff instead of custom SpaceBeforeClassBraceSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308702 [10:20:49] Yippee, build fixed! [10:20:50] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #148: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/148/ [10:25:08] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Use upstream sniff instead of custom SpaceBeforeClassBraceSniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/308702 (owner: 10Legoktm) [10:32:05] (03PS2) 10Hashar: Make EducationProgram depend on WikiEditor for tests [integration/config] - 10https://gerrit.wikimedia.org/r/303419 (owner: 10Reedy) [10:32:12] (03CR) 10Hashar: [C: 032] Make EducationProgram depend on WikiEditor for tests [integration/config] - 10https://gerrit.wikimedia.org/r/303419 (owner: 10Reedy) [10:32:34] hashar, hi, i uploaded my first patch to upstream gerrit https://gerrit-review.googlesource.com/#/c/85393/ :) :) [10:33:11] Oh wrong ling [10:33:12] link [10:33:17] (03Merged) 10jenkins-bot: Make EducationProgram depend on WikiEditor for tests [integration/config] - 10https://gerrit.wikimedia.org/r/303419 (owner: 10Reedy) [10:33:27] This one [10:33:27] https://gerrit-review.googlesource.com/#/c/85340/ [10:33:36] paladox: Please reduce your amount of comments. Your last three comments in https://phabricator.wikimedia.org/T144565 could have all been merged into one with a bit more patience. Thanks. [10:33:45] paladox: And congrats to your patch in upstream! :) [10:34:38] andre__ Your welcome, but if you look at the timestamp, there was a huge wait time, plus it was really late so i wasent really thinkining so sorry [10:34:41] :) [10:35:40] (03PS2) 10Niedzielski: Remove Android Commons app job [integration/config] - 10https://gerrit.wikimedia.org/r/308669 (https://phabricator.wikimedia.org/T127678) [10:35:44] (03PS3) 10Hashar: Remove Android Commons app job [integration/config] - 10https://gerrit.wikimedia.org/r/308669 (https://phabricator.wikimedia.org/T127678) (owner: 10Niedzielski) [10:36:36] (03CR) 10Hashar: [C: 032] "Indeed from https://gerrit.wikimedia.org/r/#/c/272212/ and T127678, the application has been forked to github and is maintained there." [integration/config] - 10https://gerrit.wikimedia.org/r/308669 (https://phabricator.wikimedia.org/T127678) (owner: 10Niedzielski) [10:37:03] stashbot seems to not be working in here [10:37:50] !log gerrit: mark apps/android/commons hidden since it is now community maintained on GitHub. Will avoid confusion. T127678 [10:37:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:37:59] (03Merged) 10jenkins-bot: Remove Android Commons app job [integration/config] - 10https://gerrit.wikimedia.org/r/308669 (https://phabricator.wikimedia.org/T127678) (owner: 10Niedzielski) [10:39:03] paladox: I can only generally warn you that more people will ignore your messages when you create too many. Just saying. :) [10:39:14] oh ok [10:39:17] sorry [10:39:44] paladox: stashbot is working, it is reporting to phabricator https://phabricator.wikimedia.org/T127678#2610565 [10:39:51] Oh [10:39:53] but the IRC component has disappeared indeed [10:39:55] not a big deal [10:39:58] Ok [10:40:02] the !log entries are recorded [10:40:05] i did those three comments at around 9:40pm, 11pm, and 11pm [10:40:08] oh [10:41:04] paladox: "Code looks OK. Will wait for other comments about whether it should go to stable-2.12." :D [10:41:04] looks great [10:41:13] Yep [10:41:25] I am going to see if they can merge in stable-2.12 [10:41:30] David Pursehouse is great based on the writings I have seen from him [10:41:42] if not i will need to ask chad if we can use that patch on our gerrit [10:41:46] and then set it to true [10:41:54] oh [10:41:56] yep [10:42:08] well [10:42:13] they are nice, and according to qchris they welcome all patches [10:42:16] maybe the fix can be backported to 2.12 yeah [10:42:22] yep [10:42:25] else we will get it when we eventually upgrade to 2.13 :] [10:42:29] I have tested it on gerrit 2.3 [10:42:31] 13 [10:42:34] 2.13 [10:42:52] Actually an upgrade to 2.13 will be really good [10:42:59] better scroll bars since there bigger [10:43:05] I am not sure it is much of a priority of us [10:43:08] and a new unified view [10:43:14] the old 2.8 was definitely a huge trouble [10:43:23] we will probably stick to 2.12 for a bit [10:43:26] Yep, but it makes diff get cutoff if there big [10:43:28] yep [10:43:38] I have http://gerrit-new.wmflabs.org/r/#/c/2/ [10:43:54] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:45:09] and looks good too me :) [10:53:13] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2610597 (10Paladox) @Tobi_WMDE_SW Hi, yes that is correct to set it read only, or do you want the repo to also be hidden. Also for h... [10:54:52] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610601 (10zeljkofilipin) Done so far: - moved everythi... [10:55:27] I accidentally used mon compte personnel with mon vrai nom and image on https://gerrit-review.googlesource.com/#/c/85355/ so i asked for it to be removed and i replaced it with https://gerrit-review.googlesource.com/#/c/85340/ :) [10:55:31] hashar ^^ [10:55:52] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2610603 (10Tobi_WMDE_SW) @Paladox thx for the hint, I've now set it to hidden since there is no point in leaving it visible IMO. [10:56:09] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2610604 (10Tobi_WMDE_SW) [10:56:21] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2610605 (10Paladox) >>! In T144486#2610603, @Tobi_WMDE_SW wrote: > @Paladox thx for the hint, I've now set it to hidden since there i... [10:56:52] 10Browser-Tests-Infrastructure, 10Gerrit, 10Wikidata, 13Patch-For-Review, 15User-Tobi_WMDE_SW: Retire wikidata/browsertests.git - https://phabricator.wikimedia.org/T144486#2610606 (10Tobi_WMDE_SW) [11:07:01] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610618 (10zeljkofilipin) Cleaned up categories: - [[ h... [11:08:47] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610619 (10zeljkofilipin) Categories that need cleaning... [11:33:12] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610669 (10Amire80) [11:34:23] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610683 (10Amire80) [11:37:01] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610669 (10Paladox... [11:42:22] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610687 (10Amire80... [11:47:58] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610669 (10Nikerabb... [11:48:21] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610669 (10KartikMi... [11:56:57] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610704 (10Nikerabb... [12:00:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610707 (10Nikerabb... [12:12:53] hashar hi https://phabricator.wikimedia.org/T144802 that seems to also be happening for mw [12:13:00] core [12:13:11] See https://gerrit.wikimedia.org/r/#/c/308329/ [12:13:18] where i did a recheck after it was merged [12:14:04] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610745 (10Paladox... [12:15:41] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610756 (10hashar) [12:16:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests fail with an error about peerDependencies - https://phabricator.wikimedia.org/T144802#2610669 (10hashar)... [12:17:18] paladox: will check [12:19:01] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 05Language-Engineering July-September 2016, and 2 others: Jenkins QUnit tests on chromium fails with "Script error." - https://phabricator.wikimedia.org/T144802#2610765 (10hashar) [12:21:10] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 4 others: Jenkins QUnit tests on chromium fails with "Script error." - https://phabricator.wikimedia.org/T144802#2610773 (10thiemowmde) [12:23:51] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 4 others: Jenkins QUnit tests on chromium fails with "Script error." - https://phabricator.wikimedia.org/T144802#2610778 (10hashar) p:05High>03Unbreak! a:03hasha... [12:25:25] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 4 others: Jenkins QUnit tests on chromium fail with "Script error." - https://phabricator.wikimedia.org/T144802#2610786 (10Amire80) [12:33:41] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 3 others: Jenkins QUnit tests on chromium fail with "Script error." - https://phabricator.wikimedia.org/T144802#2610826 (10Tobi_WMDE_SW) [12:35:18] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 3 others: Jenkins QUnit tests on chromium fail with "Script error." - https://phabricator.wikimedia.org/T144802#2610828 (10hashar) @Jonas all the qunit jobs are block... [13:00:42] thanks [13:03:05] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610899 (10zeljkofilipin) [13:05:06] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#1512435 (10zeljkofilipin) [13:07:16] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610920 (10zeljkofilipin) [13:09:19] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610921 (10zeljkofilipin) [13:09:43] hashar could the problem be because of https://github.com/wikimedia/operations-puppet/commit/09237164871afd8eda262d537fa30079fe2340c2 ? [13:11:48] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610925 (10zeljkofilipin) [13:14:33] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610929 (10zeljkofilipin) [13:19:16] (03PS1) 10Hashar: dib: explicitly include libapache-mod-php5 [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) [13:20:08] (03PS1) 10Paladox: Make sure package libapache2-mod-php5 is installed on debian jessie [integration/config] - 10https://gerrit.wikimedia.org/r/308726 (https://phabricator.wikimedia.org/T144802) [13:20:10] !log Rebuilding Nodepool Jessie image to hopefully include libapache-mod-php5 and restore qunit jobs behavior T144802 [13:20:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:20:20] hashar https://gerrit.wikimedia.org/r/308726 [13:24:12] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610951 (10zeljkofilipin) [13:28:00] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 4 others: Jenkins QUnit tests on chromium fail with "Script error." - https://phabricator.wikimedia.org/T144802#2610967 (10hashar) With the hack https://gerrit.wikime... [13:28:33] (03CR) 10Hashar: "Being rebuild:" [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) (owner: 10Hashar) [13:31:00] (03CR) 10Paladox: [C: 031] dib: explicitly include libapache-mod-php5 [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) (owner: 10Hashar) [13:31:07] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610972 (10zeljkofilipin) [13:31:15] (03Abandoned) 10Paladox: Make sure package libapache2-mod-php5 is installed on debian jessie [integration/config] - 10https://gerrit.wikimedia.org/r/308726 (https://phabricator.wikimedia.org/T144802) (owner: 10Paladox) [13:31:24] paladox: oh [13:31:32] well it is really ahack [13:31:37] oh [13:31:39] something is missing in operations/puppet.git [13:31:59] Yep, not sure what is though looking on the contint folder [13:33:35] mystery :] [13:33:55] oh [13:34:16] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2610982 (10zeljkofilipin) [13:34:19] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:34:42] waiting for the image to build [13:41:43] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2611013 (10zeljkofilipin) [13:46:09] !log nodepool.SnapshotImageUpdater: Image ci-jessie-wikimedia-1473169259 in wmflabs-eqiad is ready T144802 [13:46:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [13:47:48] Yippee, build fixed! [13:47:48] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #137: 09FIXED in 3 min 47 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/137/ [13:53:32] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 4 others: Jenkins QUnit tests on chromium fail with "Script error." - https://phabricator.wikimedia.org/T144802#2611043 (10hashar) I ran a job and it passed. I am goi... [13:53:39] (03CR) 10Hashar: [C: 032] dib: explicitly include libapache-mod-php5 [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) (owner: 10Hashar) [13:53:51] (03PS2) 10Hashar: dib: explicitly include libapache-mod-php5 [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) [13:53:54] (03CR) 10Hashar: dib: explicitly include libapache-mod-php5 [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) (owner: 10Hashar) [13:53:57] (03CR) 10Hashar: [C: 032] dib: explicitly include libapache-mod-php5 [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) (owner: 10Hashar) [13:55:52] (03Merged) 10jenkins-bot: dib: explicitly include libapache-mod-php5 [integration/config] - 10https://gerrit.wikimedia.org/r/308725 (https://phabricator.wikimedia.org/T144802) (owner: 10Hashar) [13:58:30] !log Qunit jobs should be all fine again now. T144802 [13:58:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:14:19] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:28:28] PROBLEM - Puppet staleness on deployment-salt02 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [15:12:02] 10Gerrit: Update site CSS customizations for the new change screen in Gerrit 2.12 - https://phabricator.wikimedia.org/T141286#2611228 (10Paladox) @PleaseStand bump, any updates on this please? [15:14:23] 10Gerrit: Update site CSS customizations for the new change screen in Gerrit 2.12 - https://phabricator.wikimedia.org/T141286#2611230 (10Aklapper) p:05Triage>03Low @Paladox: No suggestions and no updates (assuming that answers the question, on behalf of PleaseStand). [15:15:12] 10Gerrit: Update site CSS customizations for the new change screen in Gerrit 2.12 - https://phabricator.wikimedia.org/T141286#2611232 (10Paladox) I guess we can leave this open for another month and if no more reply's then close as resolve. [15:26:41] 10Gerrit: Raise gerrit's memory limit - https://phabricator.wikimedia.org/T141064#2611241 (10Paladox) [15:38:24] Yippee, build fixed! [15:38:25] Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #149: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/149/ [15:45:36] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #149: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/149/ [16:01:55] Yippee, build fixed! [16:01:56] Project selenium-CentralNotice » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #139: 09FIXED in 55 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/139/ [16:12:56] 10Gerrit, 13Patch-For-Review: Gerrit bug search not (naively) working with Phabricator tasks - https://phabricator.wikimedia.org/T85002#2611388 (10Paladox) The above change requires a reindex. [16:16:53] 10Gerrit, 07Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#403845 (10Paladox) @Nikerabbit hashtag is a new feature and is in gerrit 2.13, not gerrit 2.12. But I thought this task is about topic. [16:23:28] 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog, 07Browser-Tests, 03Reading-Web-Sprint-81-W: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#2611425 (10MBinder_WMF) [16:42:18] RECOVERY - Host deployment-parsoid05 is UP: PING OK - Packet loss = 0%, RTA = 0.52 ms [16:43:41] ostriches can we include this patch https://gerrit-review.googlesource.com/#/c/85340/ in the gerrit update please? With another minor change for our install to turn it on by default [16:43:55] it wont break anything, it just adds a new pref [16:44:26] No, I don't pull in custom commits. [16:44:44] ok [16:44:46] Especially if it's not merged. [16:45:17] ostriches upstream say it looks good, but doint know weather to include it in stable-2.12 [16:45:38] If it's not going into 2.12 then no [16:45:50] Ok, it fixes https://phabricator.wikimedia.org/T144565 [16:46:03] That doesn't change my answer. [16:46:08] ok [16:51:45] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [16:55:36] PROBLEM - Puppet run on deployment-kafka05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [17:08:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Translate, 10MediaWiki-extensions-UniversalLanguageSelector, 10Wikidata, and 4 others: Jenkins QUnit tests on chromium fail with "Script error." - https://phabricator.wikimedia.org/T144802#2611667 (10greg) 05Open>03Resolved >>! In T144802#... [17:12:37] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:30:37] RECOVERY - Puppet run on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:35:35] 10Browser-Tests-Infrastructure, 06Release-Engineering-Team, 10MediaWiki-extensions-Examples, 07Documentation, and 2 others: Improve documentation around running/writing (with lots of examples) browser tests - https://phabricator.wikimedia.org/T108108#2611790 (10zeljkofilipin) [17:52:37] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [18:10:38] twentyafterfour_ could you review https://gerrit.wikimedia.org/r/#/c/308308/ please? [18:11:01] twentyafterfour ^^ [18:11:12] oh seems he is not named twentyafterfour_ now sorry [18:17:30] jenkins is slow again [18:17:31] https://integration.wikimedia.org/zuul/ [18:17:41] only 3 tests being tested again [18:17:48] hasharAway ^^ [18:20:32] greg-g ^^ chasemp [18:20:40] Looks like nodepool again [18:32:59] more than likely it is https://graphite.wikimedia.org/render/?width=2064&height=1100&_salt=1472565940.394&target=cactiStyle(nodepool.job.*trusty*.runtime.mean)&hideLegend=false&from=-2h&lineMode=connected [18:33:16] trusty jobs taking a long time to run, which isn't nodepool's issue directly [18:33:24] but why jobs take so long all of a sudden I have no idea [18:33:31] someoen with more knowledge of the jobs would have to look [18:33:45] but that does prevent instance turnover and causes backlog [18:34:40] ^ hasharAway greg-g thcipriani trusty jobs taking a long time to run [18:34:46] jumped about 10 after or 30m ago [18:35:11] hrm [18:36:07] fwiw atm there is 4 for trusty, 4 for jessie min-ready and 4 slush for either nd trusty is just absorbing it all and hanging out [18:36:11] and so jessie jobs are also affected [18:36:24] https://graphite.wikimedia.org/render/?width=966&height=489&_salt=1471549160.573&target=cactiStyle(zuul.pipeline.gate-and-submit.label.ci*.wait_time.mean)&hideLegend=false&lineMode=connected&from=-6h [18:43:42] thcipriani: did you do something? [18:44:04] chasemp: nope still digging through jobs/graphs [18:51:41] https://graphite.wikimedia.org/render/?width=2064&height=1100&_salt=1472565940.394&areaMode=stacked&target=nodepool.job.*trusty*.runtime.mean&hideLegend=false&mode=stacked&from=-2h [18:51:51] https://graphite.wikimedia.org/render/?width=2064&height=1100&_salt=1472565940.394&areaMode=stacked&target=nodepool.job.*jessie*.runtime.mean&hideLegend=false&mode=stacked&from=-2h [18:51:53] thcipriani: ^ [18:52:02] view I like of job runtimes that kind of outlines [18:52:54] we just are not setup for that kind of load and length of runtime for jobs [18:53:04] chasemp: yeah, that's neat [18:53:17] seems like some of these test runtimes are pretty noisy [18:53:29] over time: https://graphite.wikimedia.org/render/?width=2064&height=1100&_salt=1472565940.394&target=cactiStyle(nodepool.job.*-trusty*.runtime.mean)&hideLegend=false&from=-2w&lineMode=connected [18:54:23] mediawiki-core-phpcs-trusty [18:54:32] we reverted what triggers that last week iirc [18:54:40] everytime it batters the setup [18:54:50] I can't for the life of me figure out how things were running along before [18:55:10] was it unstable and prone to huge spikes in wait times? [18:57:31] beats me? Hadn't noticed that job doing anything crazy recently. [18:57:48] probably says more about my general awareness of individual CI jobs than anything else [19:09:30] chasemp: thcipriani o/ [19:13:44] the snapshot images were fairly outdated and I have rebuild them from scratch earlier today [19:13:58] the jessie one lacked libapache2-mod-php5 which broke the qunit jobs ( fixed ) [19:14:04] the trusty ones might have some random other issue [19:15:29] oh, that might explain it. I'm not sure if some of these trusty jobs occasionally running slowly is outside the norm. [19:18:21] https://grafana.wikimedia.org/dashboard/db/releng-zuul usually has much of the metrics [19:18:31] the gearman queue is a good indicator of what is waiting [19:18:48] it had a slight spike at 18:05 [19:19:26] my theory is [19:19:46] the pool is 12 instances. 4 were jessie, 8 for trusty [19:19:57] with most trusty one being busy with 10+ minutes long jobs [19:21:35] yes atm there is no way for us to sustain thsoe kinds of runtimes on a job since all jobs are blocking etc [19:21:41] with reasonable waits I mean [19:22:17] we used to have a few big instances running up to 5 jobs in parallel [19:22:23] but that eventually caused a bunch of troubles [19:22:43] so the 15 or so permanent Trusty slaves now only have 2 jobs in parallel each [19:22:46] or 30 executors [19:28:20] hashar: thcipriani it occurs to me we can't have a meaningful sla on wait-time unless we have a defined max-test-run-time as well, or at least the two are interdependent [19:29:32] yeah they are interdependant [19:29:46] the longer the tests are run, the longest the pool is busy / unable to process other jobs [19:30:04] a typical denial of service we have is realeasing new version of mediawiki versions [19:30:27] the last one was terrible asking for roughly 600 jobs over a couple minutes [19:32:03] can we gate that by job type so that it doesn't affect say ops/puppet linting? [19:32:15] that's the real issue, there is no QoS for jobs [19:32:47] the pipelines (eg: test, gate-and-submit) can receive a precedence [19:32:56] with gate-and-submit being high prio [19:33:29] basically Zuul submit the jobs to Gearman with the high prio bit set [19:33:30] we've talked about dividing jobs into pipelines per-branch, that may help some, and seems to make sense with zuul's mode of operation [19:33:36] and Gearman will process them in priorty [19:35:01] it seems like to do any sort of planning we'll have to do that [19:43:55] I am digging in nodepool logs [19:45:11] 2016-09-06 18:27:33 Manager wmflabs-eqiad running "CreateServerTask" (queue: 4) [19:47:29] that is not a good example sorry [19:47:31] but in short [19:47:41] the server creation requests are serialized in the global queue [19:47:50] which is processed at 1 event every 8 seconds [19:48:39] so if we have a surge of jobs, it takes some time to process the deletion tasks (to free up the quota) and then process the instances creation tasks [19:50:22] and the tasks do not track their age [19:50:31] so no idea how long they have been waiting [19:52:07] why would that matter? [19:55:56] 10scap, 06Operations, 06Performance-Team, 07Epic: During deployment old servers may populate new cache URIs - https://phabricator.wikimedia.org/T47877#2612381 (10Krinkle) [20:03:38] chasemp: that is taken in account by the wait_time [20:04:06] can you break that down specifically, I mean why would tasks not tracking their own age matter [20:04:12] Nodepool notice 1 server is needed. It creates a task and push it to the queue [20:04:15] I assume it's metadata tracked by teh system and thus wait-time [20:04:37] if the queue has 10 items to process, that will take 80 seconds just for API call to be made [20:05:26] I have added a "Tasks" graph to https://grafana-admin.wikimedia.org/dashboard/db/nodepool [20:05:28] sure I understand 8*10 man, we don't know what rate we can sustain because we've never done it in a stable manner [20:05:41] ideally we just make it as fast as possible? [20:05:46] but w/ tests that take 10m [20:05:51] well rate has been at 1 second until August 15th [20:05:56] it doesn't seem like the differentiator [20:06:00] sure and things were a mess? [20:06:10] on our side at least [20:06:16] not so much though you discovered at least a couple issue on the infra [20:06:23] so yeah that was not so good [20:06:40] the reason for going to 1 second was to speed up the deletion/creation [20:06:48] sure that doesn'tn work tho [20:06:49] but we haven't looked at the side effects it would cause on the labs infra [20:06:52] that just speeds up requests [20:06:54] (or at least I did not ) [20:07:02] and requets are not actual work in this case [20:07:09] yeah [20:07:11] as demonstrated by greater than 50% 403's [20:07:35] if changing an 8 to a 1 was meaningful I would just do it [20:07:47] but we had to up the db connection pool for just 2 of 5 reverts last week [20:07:57] so I'm not sure we can definitely say all load historically present makes sense [20:08:03] yeah [20:08:38] would it be possible that the SqlAlchemy pool being to short has been a long standing issue tht never got noticed? [20:09:01] I'm sure it was [20:09:25] it was mainly made easier to find w/ the quota adjustments happening constantly and using up a set amount of connections [20:09:42] but that isn't enough load to say it wasn't happening for a long time at various amounts of activity [20:11:57] anyone can review https://gerrit.wikimedia.org/r/#/c/299697/ please? [20:16:15] chasemp: so should I still hold the reverts or can we bring on some more jobs ? [20:16:45] (03PS2) 10Legoktm: Publish Doxygen and jsduck documentation for Kartographer [integration/config] - 10https://gerrit.wikimedia.org/r/299697 (https://phabricator.wikimedia.org/T140657) (owner: 10MaxSem) [20:17:06] do we know how the remaining 3 stack up proportionally against hte 2 we did hashar? [20:17:21] MaxSem: will do [20:17:36] oh [20:17:54] MaxSem: well legoktm has rebased your change for Kartographer so I guess he is going to push it :] [20:19:47] chasemp: I crafted a Grafana board to find out the metric based on the job names [20:19:49] (03CR) 10Legoktm: [C: 04-1] "Kartographer doesn't have a Doxyfile?" [integration/config] - 10https://gerrit.wikimedia.org/r/299697 (https://phabricator.wikimedia.org/T140657) (owner: 10MaxSem) [20:19:59] chasemp: for the ruby/rake jobs that is roughly 300 builds per days https://grafana.wikimedia.org/dashboard/db/zuul-job?var-pipeline=All&var-job=*rake*&var-status=All [20:20:17] 1 build == 1 job? [20:21:31] (== 1 tied up nodepool instance) [20:21:36] yes [20:21:49] is that jessie? [20:21:55] yes sir [20:22:18] want it try to move that first thing tomorrow? [20:23:11] maybe that is the best candidate after all [20:23:16] looks like a lot of jobs [20:23:24] yeah it's pretty heavy [20:23:57] the tox jobs are half the payload or roughly 150 jobs [20:25:08] (03PS1) 10Hoo man: Update Wikidata to the wmf/1.28.0-wmf.18 branch [tools/release] - 10https://gerrit.wikimedia.org/r/308827 [20:25:48] chasemp: lets do the tox jobs tomorrow [20:25:56] what revert is that? [20:25:59] i will prepare / rebase the CI changes and get them ready to push whenever you show up [20:26:04] tox [20:26:04] looking at the task and am unsure [20:26:17] on the table here I mean https://phabricator.wikimedia.org/T143938 [20:26:28] hmm [20:26:29] wait [20:26:34] I am lost [20:26:42] we did them already :] [20:26:42] (03CR) 10Chad: [C: 032 V: 032] Update Wikidata to the wmf/1.28.0-wmf.18 branch [tools/release] - 10https://gerrit.wikimedia.org/r/308827 (owner: 10Hoo man) [20:26:49] yeah I was confused there [20:29:09] 10Continuous-Integration-Infrastructure, 07Nodepool, 13Patch-For-Review: Bring back jobs to Nodepool - https://phabricator.wikimedia.org/T143938#2612591 (10Legoktm) Before we move the composer jobs back, I'd like to implement {T142457}. [20:31:54] chasemp: yeah the rake jobs will be next definitely [20:32:11] timo made it so the job is only triggered if a change alter a .rb file [20:32:15] which item in the table is that? [20:32:15] or gemfile or whatever related [20:32:25] https://gerrit.wikimedia.org/r/#/c/306723/ Revert "Move rake jobs off of nodepool" [20:32:28] https://gerrit.wikimedia.org/r/#/c/306724/ Revert "rake: Fix bundle install path" [20:32:52] so how many jobs per day? [20:32:54] and looking at the # of builds https://grafana.wikimedia.org/dashboard/db/zuul-job?var-pipeline=All&var-job=*rake*&var-status=All [20:33:13] the monthly graph bottom right show it was triggered around 300 times per day [20:33:27] but since Sep 1th or so, it is only a few builds [20:33:34] as shown on the weekly view (bottom left) [20:33:58] given the # of builds, is really just background noise :] [20:34:29] ok marking it down to work out that tomorrow and then we'll see what happens [20:35:48] chasemp: I will do it in my morning [20:36:04] and prepare the next batch [20:38:20] Project performance-webpagetest-wmf build #2507: 04FAILURE in 1 hr 55 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wmf/2507/ [20:40:17] 10Continuous-Integration-Infrastructure, 07Nodepool, 13Patch-For-Review: Bring back jobs to Nodepool - https://phabricator.wikimedia.org/T143938#2612633 (10hashar) The tox jobs are all happy. ---- The rake jobs have been made to only trigger when ruby related files are modified (T144325). It is done to on... [21:04:55] https://www.fastcompany.com/3062835/hr/microsoft-autism-hiring :) [22:02:39] Hi. [22:02:47] bd808: there is no project for stashbot? [22:02:50] (on Phab) [22:03:08] Dereckson: nope. [22:03:38] you can file something under tool-labs-other and assign to me if you'd like [22:04:33] Done. [22:07:08] Dereckson: lol. yeah that would be annoying [22:43:44] 10Continuous-Integration-Infrastructure: Support PHP 7 in CI infra - https://phabricator.wikimedia.org/T144872#2613127 (10Legoktm) [23:04:53] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [23:10:38] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T142855#2613211 (10hoo) [23:31:37] Yippee, build fixed! [23:31:38] Project performance-webpagetest-wmf build #2508: 09FIXED in 1 hr 48 min: https://integration.wikimedia.org/ci/job/performance-webpagetest-wmf/2508/ [23:44:55] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:46:53] RoanKattouw Hi, i got around to fixing gerrit diffs, see https://gerrit-review.googlesource.com/#/c/85340/ please [23:47:02] also tested on https://gerrit-new.wmflabs.org/ [23:47:21] and it seems pre-wrap now works too, and you can properly select text too [23:48:57] Nice! [23:49:46] RoanKattouw its also a new pref, so you will need to go into the diff edit settings bit to enable it. [23:53:31] :)