[00:11:18] RECOVERY - Puppet errors on deployment-maps01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:17:14] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4052182 (10awight) I think deployment-tin and tin will need git-lfs. [00:27:04] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4052212 (10awight) I must be missing something, scap deploys the files without LFS hydrating them: ``` awight@deployment-ores01 cd /srv/deployment/ores/deploy/submodules/assets/ cat test.bin.gz version https://git-lfs.gi... [00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:46:44] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4052240 (10Paladox) Set git_binary_manager: git-lfs In scap.cfg [00:55:31] did something change with selenium recently? I get increase in random failures in builds, e.g.: https://integration.wikimedia.org/ci/job/mwext-mw-selenium-composer-jessie/9136/console [01:24:40] PROBLEM - Free space - all mounts on deployment-tin is CRITICAL: CRITICAL: deployment-prep.deployment-tin.diskspace._mnt.byte_percentfree (No valid datapoints found)deployment-prep.deployment-tin.diskspace._srv.byte_percentfree (<30.00%) WARN: deployment-prep.deployment-tin.diskspace.root.byte_percentfree (<10.00%) [02:52:29] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [04:30:14] (03CR) 10Chad: [V: 032 C: 032] Grant Edit hashtag right to everyone. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/416357 (https://phabricator.wikimedia.org/T37534) (owner: 10Paladox) [06:00:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikidata: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4052385 (10Smalyshev) [06:49:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikidata: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4052421 (10Smalyshev) Same happens with `mediawiki-extensions-qunit-jessie`. [06:56:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikidata: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4052385 (10Legoktm) >>! In T189762#4052421, @Smalyshev wrote: > Same happens with `mediawiki-extensions-qunit-jessie`. That's {T189329}. [07:33:04] (03CR) 10Hashar: [C: 032] "Surely later on we can come with a job that retrieves the artifacts with git-lfs and maybe run tests of ores using those :]" [integration/config] - 10https://gerrit.wikimedia.org/r/419610 (https://phabricator.wikimedia.org/T180627) (owner: 10Awight) [07:34:14] (03Merged) 10jenkins-bot: Automatically merge on a new repo [integration/config] - 10https://gerrit.wikimedia.org/r/419610 (https://phabricator.wikimedia.org/T180627) (owner: 10Awight) [07:48:42] (03PS1) 10Hashar: Use a Zuul mutex for the coverage patch jobs [integration/config] - 10https://gerrit.wikimedia.org/r/419674 (https://phabricator.wikimedia.org/T72597) [07:54:06] (03CR) 10Hashar: [C: 032] "Giving it a try. I noticed yesterday some build could not be done because the node got deleted by the time the job had a chance to run." [integration/config] - 10https://gerrit.wikimedia.org/r/419674 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [07:55:30] (03Merged) 10jenkins-bot: Use a Zuul mutex for the coverage patch jobs [integration/config] - 10https://gerrit.wikimedia.org/r/419674 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [08:48:35] (03CR) 10Legoktm: "This looks exciting :)" [integration/config] - 10https://gerrit.wikimedia.org/r/419674 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [08:51:25] (03CR) 10Legoktm: "I think this is the third example in " [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419126 (owner: 10Thiemo Kreuz (WMDE)) [08:52:27] 10Release-Engineering-Team (Kanban): mediawiki-core-code-coverage & mediawiki-core-code-coverage-php7 jobs lock up labs integration slaves - https://phabricator.wikimedia.org/T189693#4052527 (10hashar) a:05hashar>03None Yesterday I have moved the phpunit-coverage-patch jobs to Nodepool ( [[ https://gerrit.wi... [08:52:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): mediawiki-core-code-coverage & mediawiki-core-code-coverage-php7 jobs lock up labs integration slaves - https://phabricator.wikimedia.org/T189693#4052531 (10hashar) [08:54:44] (03PS3) 10Legoktm: Validate phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/418732 (https://phabricator.wikimedia.org/T176031) (owner: 10Umherirrender) [08:54:46] (03PS2) 10Legoktm: Skip closing parentheses in "( )" and "[ ]" instead of rechecking [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419149 (owner: 10Thiemo Kreuz (WMDE)) [08:54:48] (03PS2) 10Legoktm: Check for close parenthesis first and shorten out earlier [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419150 (owner: 10Thiemo Kreuz (WMDE)) [08:54:50] (03PS2) 10Legoktm: Skip empty () and [], not processing closing token a second time [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419152 (owner: 10Thiemo Kreuz (WMDE)) [08:55:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban): mediawiki-core-code-coverage & mediawiki-core-code-coverage-php7 jobs lock up labs integration slaves - https://phabricator.wikimedia.org/T189693#4052535 (10hashar) @addshore wrote: > Offset the coverage jobs so then run at different... [08:55:29] (03CR) 10Legoktm: [C: 032] "Manually rebased." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/418732 (https://phabricator.wikimedia.org/T176031) (owner: 10Umherirrender) [08:55:48] (03CR) 10Legoktm: [C: 032] Skip closing parentheses in "( )" and "[ ]" instead of rechecking [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419149 (owner: 10Thiemo Kreuz (WMDE)) [08:56:00] (03CR) 10Legoktm: [C: 032] Check for close parenthesis first and shorten out earlier [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419150 (owner: 10Thiemo Kreuz (WMDE)) [08:56:14] (03Merged) 10jenkins-bot: Validate phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/418732 (https://phabricator.wikimedia.org/T176031) (owner: 10Umherirrender) [08:56:34] (03Merged) 10jenkins-bot: Skip closing parentheses in "( )" and "[ ]" instead of rechecking [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419149 (owner: 10Thiemo Kreuz (WMDE)) [08:56:39] (03CR) 10jenkins-bot: Validate phpunit annotations [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/418732 (https://phabricator.wikimedia.org/T176031) (owner: 10Umherirrender) [08:56:43] (03CR) 10Legoktm: [C: 032] Skip empty () and [], not processing closing token a second time [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419152 (owner: 10Thiemo Kreuz (WMDE)) [08:57:02] (03CR) 10jenkins-bot: Skip closing parentheses in "( )" and "[ ]" instead of rechecking [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419149 (owner: 10Thiemo Kreuz (WMDE)) [08:57:07] tgr reported that gerrit email are backlogged in -devtools, does the email work for you hashar or legoktm? [08:57:09] (03CR) 10Hashar: [C: 032] "Maybe :] I noticed the issue yesterday night and this morning after I had moved the patch coverage jobs to Nodepool." [integration/config] - 10https://gerrit.wikimedia.org/r/419674 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [08:58:17] Email works for me [08:58:33] (03PS3) 10Legoktm: Add missing newlines to multi-line if() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419126 (owner: 10Thiemo Kreuz (WMDE)) [08:58:47] (03CR) 10Legoktm: [C: 032] "..." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419150 (owner: 10Thiemo Kreuz (WMDE)) [08:59:04] (03CR) 10Legoktm: [C: 032] "..." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419152 (owner: 10Thiemo Kreuz (WMDE)) [08:59:37] (03Merged) 10jenkins-bot: Check for close parenthesis first and shorten out earlier [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419150 (owner: 10Thiemo Kreuz (WMDE)) [08:59:40] (03CR) 10Legoktm: [C: 032] "Manually rebased" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419126 (owner: 10Thiemo Kreuz (WMDE)) [09:00:08] (03CR) 10jenkins-bot: Check for close parenthesis first and shorten out earlier [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419150 (owner: 10Thiemo Kreuz (WMDE)) [09:05:09] (03PS1) 10Hashar: gate-and-submit must reject merged changes [integration/config] - 10https://gerrit.wikimedia.org/r/419686 (https://phabricator.wikimedia.org/T188290) [09:05:29] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: User commenting on a merge change that it had Cr+2 cause it to enter gate-and-submit again - https://phabricator.wikimedia.org/T188290#4052562 (10hashar) [09:06:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: User commenting on a merge change that it had Cr+2 cause it to enter gate-and-submit again - https://phabricator.wikimedia.org/T188290#4002561 (10hashar) p:05Triage>03Normal [09:06:13] (03CR) 10jerkins-bot: [V: 04-1] gate-and-submit must reject merged changes [integration/config] - 10https://gerrit.wikimedia.org/r/419686 (https://phabricator.wikimedia.org/T188290) (owner: 10Hashar) [09:24:23] !log deployment-prep: added gehel and dcausse has members/admins of the labs project [09:24:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:30:12] thanks :) [09:56:22] (03PS2) 10Hashar: gate-and-submit must reject merged changes [integration/config] - 10https://gerrit.wikimedia.org/r/419686 (https://phabricator.wikimedia.org/T188290) [09:57:27] (03CR) 10jerkins-bot: [V: 04-1] gate-and-submit must reject merged changes [integration/config] - 10https://gerrit.wikimedia.org/r/419686 (https://phabricator.wikimedia.org/T188290) (owner: 10Hashar) [10:03:56] (03PS3) 10Hashar: Migrate VisualEditor screenshot job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/416958 (https://phabricator.wikimedia.org/T189122) [10:26:26] (03PS4) 10Hashar: Migrate VisualEditor screenshot job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/416958 (https://phabricator.wikimedia.org/T189122) [10:30:44] (03PS5) 10Hashar: Migrate VisualEditor screenshot job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/416958 (https://phabricator.wikimedia.org/T189122) [10:38:57] (03PS17) 10Zoranzoki21: Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [10:40:02] (03CR) 10jerkins-bot: [V: 04-1] Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) (owner: 10Zoranzoki21) [10:41:19] (03PS6) 10Hashar: Migrate VisualEditor screenshot job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/416958 (https://phabricator.wikimedia.org/T189122) [10:47:34] (03CR) 10Hashar: [C: 04-1] "Dependencies got merged. A run is on going at https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor/BROWSER=chrome,PL" [integration/config] - 10https://gerrit.wikimedia.org/r/416958 (https://phabricator.wikimedia.org/T189122) (owner: 10Hashar) [10:54:07] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:52] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [11:23:04] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4052761 (10Aklapper) >>! In T188149#4052000, @JAufrecht wrote: > `Data error for transaction PHID-XACT-TASK-j4goxjs7ynylrym: no keys. Skippin... [11:46:33] 10Project-Admins, 10MediaWiki-extensions-WikimediaBadges, 10User-MarcoAurelio: Create Phabricator tag for Extension:WikimediaBadges - https://phabricator.wikimedia.org/T189373#4052809 (10Aklapper) CC'ing its maintainers (I assume, as not listed on https://www.mediawiki.org/wiki/Developers/Maintainers ). [11:57:40] (03CR) 10Thiemo Kreuz (WMDE): [C: 032] "Do we need to poke Jenkins here?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419152 (owner: 10Thiemo Kreuz (WMDE)) [11:58:24] (03Merged) 10jenkins-bot: Skip empty () and [], not processing closing token a second time [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419152 (owner: 10Thiemo Kreuz (WMDE)) [11:59:12] (03CR) 10jenkins-bot: Skip empty () and [], not processing closing token a second time [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419152 (owner: 10Thiemo Kreuz (WMDE)) [12:28:25] (03CR) 10Hashar: "Sorry it took me a while. In short:" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/415769 (https://phabricator.wikimedia.org/T160943) (owner: 10Smalyshev) [12:30:53] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:45:28] Nikerabbit: is Petar on IRC? could not find him in #mediawiki-i18n [12:45:47] https://integration.wikimedia.org/ci/blue/organizations/jenkins/beta-scap-eqiad/detail/beta-scap-eqiad/199660/pipeline [12:45:50] something wrong there? [12:58:27] no_justification: I think I may have found the closest place we should be looking at [12:58:29] For gitiles [12:58:31] https://github.com/GerritCodeReview/gitiles/blob/65edbe49f2b3882a5979f602383ef0c7b2b8ee0c/java/com/google/gitiles/Linkifier.java#L63 [12:58:33] See ^^ [12:59:53] Yes it’s that class I think [13:00:17] Linkifier sounds like it makes the links in the ui [13:01:38] Uh never mind [13:01:46] I just read the whole thing [13:01:56] It makes the comment I’d a link [13:03:08] Hmm what about https://github.com/GerritCodeReview/gitiles/blob/65edbe49f2b3882a5979f602383ef0c7b2b8ee0c/java/com/google/gitiles/PathServlet.java#L611 [13:03:38] 10Project-Admins, 10MediaWiki-extensions-WikimediaBadges, 10User-MarcoAurelio: Create Phabricator tag for Extension:WikimediaBadges - https://phabricator.wikimedia.org/T189373#4053008 (10hoo) >>! In T189373#4040362, @Aklapper wrote: > Are the maintainers of #mediawiki-extensions-wikimediabadges aware of this... [13:04:36] Or https://github.com/GerritCodeReview/gitiles/blob/84286b7deabe8178f424c490f0c011c40c8851eb/java/com/google/gitiles/GitilesFilter.java#L333 [13:08:13] https://github.com/GerritCodeReview/gitiles/blob/65edbe49f2b3882a5979f602383ef0c7b2b8ee0c/java/com/google/gitiles/DefaultUrls.java#L51 [13:08:20] Or that ^^ [13:12:57] https://github.com/GerritCodeReview/plugins_gitiles/blob/22e69dbcd6a6ac3c254019184c4b5f335f6d6ec4/src/main/java/com/googlesource/gerrit/plugins/gitiles/Module.java#L76 [13:14:09] no_justification: I found it ^^ [13:14:17] It’s in the hostName config [13:17:16] zeljkof: he is not as far as I know, what's up? [13:17:42] Nikerabbit: I need a quick check on something I translated to Serbian :) [13:18:08] zeljkof: you could try pinging him on hangouts/telegram [13:18:16] 10Gerrit, 10Release-Engineering-Team (Next), 10DBA, 10Operations, 10Patch-For-Review: Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working - https://phabricator.wikimedia.org/T176532#4053048 (10Paladox) [13:18:48] Nikerabbit: will do; by the way, he is on at the contact list at office wiki [13:18:55] is that on purpose? [13:20:36] zeljkof: "he is not"? [13:21:06] Nikerabbit: sorry, in the middle of swat :) yes, he is not there [13:21:52] zeljkof: I don't think it is on purpose. You or I can tell him to add himself there. [13:22:14] Nikerabbit: I can't find his contact data ;) [13:23:12] zeljkof: hangouts have search, right? and tg is @Pl217 [13:23:50] Nikerabbit: I don't use hangouts much, will try [14:27:34] Project mwext-phpunit-coverage-publish build #2215: 04FAILURE in 59 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2215/ [14:31:56] Yippee, build fixed! [14:31:57] Project mwext-phpunit-coverage-publish build #2216: 09FIXED in 1 min 38 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2216/ [14:45:03] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, and 3 others: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4053238 (10zeljkofilipin) How is this possibl... [14:53:16] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:55:02] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, and 3 others: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4053264 (10zeljkofilipin) Looks like commons... [15:07:47] (03PS4) 10Pablo Grass (WMDE): Run node selenium tests as part of WikibaseLexeme CI build [integration/config] - 10https://gerrit.wikimedia.org/r/419479 (https://phabricator.wikimedia.org/T189677) (owner: 10WMDE-leszek) [15:10:26] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, and 3 others: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4053293 (10zeljkofilipin) The error message s... [15:18:11] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, and 3 others: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4053300 (10zeljkofilipin) I took a quick look... [15:18:56] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, and 3 others: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4053314 (10zeljkofilipin) About `fileexists-s... [15:57:03] no_justification: when you are around please clue me in on the mwscript whch version of php it's gonna use thing [15:57:18] hardcoded back to php5 :\ [15:57:19] I need to watch this both for existing hosts and for the move of dumps to stretch+php7 [15:57:20] Hard to revert [15:57:25] *had to [15:57:44] if I want it to use php7 on some host, what would i have to do? [15:57:59] (eg snapshot01 instance in beta, or later actual prod stretch boxes) [15:58:18] note they will have hhvm installed along side of php7 and /etc/alternative/php will be pointing to that [15:58:23] to hhvm I mean [15:58:33] because appserver move to stretch will continue to use hhvm for some time [16:00:08] maybe I can avoid it, I think we mostly use multiversion/MWScript.php [16:05:31] apergos: You can override it with an environment variable [16:05:38] PHP=php7 mwscript ... [16:05:45] good enough if we need it, thanks [16:07:58] no_justification: great news I think upstream have a change to fix T161206 [16:07:59] T161206: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206 [16:08:01] :) [16:08:10] I spoke with Dave in #gerrit [16:08:39] As I was asking a question about notedb, then I showed him my proposed fix, which he made it cleaner and uploaded it here: [16:08:53] https://gerrit-review.googlesource.com/c/gerrit/+/166310 [16:11:36] 10Gerrit, 10Analytics-Tech-community-metrics, 10Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#4053473 (10Paladox) @TerraCodes nope, I spoke with Dave from google in #gerrit about this task who kindly had a look at my propose change... [16:12:16] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4053475 (10awight) Now I get the official error saying that git-lfs isn't installed on deployment-ores01. How are we doing this for other machines? git-lfs isn't in normal Debian packages. ``` 15:48:38 [deployment-ores01.... [16:14:10] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4053485 (10Paladox) @awight Is deployment-ores* stretch? You could apt-get install git-lfs if it is a stretch host. [16:16:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Install the BlueOcean plugin on jenkins - https://phabricator.wikimedia.org/T155840#4053487 (10hashar) 05Open>03Resolved a:03hashar Installed and at least @thcipriani likes that BlueOcean skin :] [16:16:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Jenkins: Install the BlueOcean plugin on jenkins - https://phabricator.wikimedia.org/T155840#4053492 (10Paladox) Ah thanks @hashar :) [16:20:23] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4053512 (10awight) @Paladox No, it's not, it's Jessie. Good catch! We'll reimage. [16:21:32] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4053523 (10Paladox) Your welcome ;) [16:36:07] !log deleting deployment-ores01 to reimage [16:36:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:40] PROBLEM - Host deployment-ores01 is DOWN: CRITICAL - Host Unreachable (10.68.23.165) [16:42:03] (03PS1) 10Hashar: Port mw-install-sqlite to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 [16:42:25] (03CR) 10jerkins-bot: [V: 04-1] Port mw-install-sqlite to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 (owner: 10Hashar) [16:43:18] !log created deployment-ores01 to use stretch [16:43:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:47:45] no_justification im wondering could we cherry pick this https://gerrit-review.googlesource.com/c/gerrit/+/166310 so we can try it. It will not break anything [16:47:51] but will confirm if it fixes our task [16:48:01] as there is no way to reproduce that issue locally [16:48:07] please [16:48:47] I'd have to rebuild and deploy it :\ [16:50:42] oh [16:51:24] no_justification with git-lfs, it should make your life easier in the future (once it deployed to all of deployment servers :)) [16:54:17] RECOVERY - Host deployment-ores01 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [16:56:02] I'd rather deploy 2.14.7 first :) [16:57:00] ok :) [16:58:42] no_justification i did this https://gerrit.wikimedia.org/r/419790 for when ever you've done 2.14.7 :) [17:01:22] 10Gerrit, 10Analytics-Tech-community-metrics, 10Upstream: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#4053759 (10Paladox) https://gerrit.wikimedia.org/r/#/c/419790/ [17:09:20] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10VisualEditor, and 3 others: Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers - https://phabricator.wikimedia.org/T189122#4053802 (10zeljkofilipin) I have updated Ruby... [17:14:17] PROBLEM - Host deployment-ores01 is DOWN: CRITICAL - Host Unreachable (10.68.19.237) [17:15:28] Nikerabbit, zeljkof - Hallo. Any idea why does https://gerrit.wikimedia.org/r/#/c/419699/ fail? [17:15:29] !log Update mobileapps to c5e1522 [17:15:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:15:36] I can't decypher the console output at all. [17:16:46] aharoni: I think it's a known problem :( there should be a task about it [17:17:15] zeljkof: ack. Thanks. It's odd that this patch failes repeatedly, but other patches in the same extensions pass. [17:17:34] aharoni: ah, maybe then it's just a failing test ;) [17:18:45] zeljkof: but I cannot decypher the output, and it doesn't seem to come from this extension (UniversalLanguageSelector). It seems to come from Wikibase. (Why oh why does Wikibase fail other extensions so frequently?) [17:19:15] aharoni: sorry, I'm not familiar with url, wikibase or qunit at all :( [17:19:17] It has "base" in it's name, but it's not a very good explanation ;) [17:24:20] RECOVERY - Host deployment-ores01 is UP: PING OK - Packet loss = 0%, RTA = 2.17 ms [17:25:40] zeljkof: https://phabricator.wikimedia.org/T189800 FWIW [17:45:08] 10Beta-Cluster-Infrastructure, 10Community-Tech, 10MediaWiki-extensions-GlobalPreferences, 10Patch-For-Review: Deploy GlobalPreferences on beta cluster - https://phabricator.wikimedia.org/T184668#4054014 (10Niharika) [17:45:59] PROBLEM - Host deployment-ores01 is DOWN: CRITICAL - Host Unreachable (10.68.18.88) [17:46:46] 10MediaWiki-Codesniffer, 10Patch-For-Review: Add sniff to ensure doc blocks don't use "@cover", but "@covers" instead - https://phabricator.wikimedia.org/T176031#4054021 (10Umherirrender) 05Open>03Resolved a:03Umherirrender [17:54:20] RECOVERY - Host deployment-ores01 is UP: PING OK - Packet loss = 0%, RTA = 0.93 ms [18:07:26] 10Release-Engineering-Team (Kanban), 10Vector, 10MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)), 10Patch-For-Review, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#4054069 (10zeljkofilipin) @Jdlrobson... [18:09:01] 10Phabricator, 10Wikimedia-Hackathon-2018: Build a bot that pushes Phabricator updates to Google Chat - https://phabricator.wikimedia.org/T189313#4054071 (10dbarratt) a:03dbarratt [18:12:47] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4054081 (10JAufrecht) You probably can't find it in the code because ... it's temporary error-handling code that I added to the dev server an... [18:20:54] RECOVERY - Puppet errors on deployment-ores01 is OK: OK: Less than 1.00% above the threshold [0.0] [18:28:36] (03CR) 10Umherirrender: [C: 032] Add missing newlines to multi-line if() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419126 (owner: 10Thiemo Kreuz (WMDE)) [18:29:25] (03Merged) 10jenkins-bot: Add missing newlines to multi-line if() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419126 (owner: 10Thiemo Kreuz (WMDE)) [18:29:55] (03CR) 10jenkins-bot: Add missing newlines to multi-line if() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419126 (owner: 10Thiemo Kreuz (WMDE)) [18:31:38] 10Continuous-Integration-Infrastructure, 10Front-end-Standards-Group, 10MediaWiki-extensions-General: Decide whether we want the package-lock.json to commit or ignore - https://phabricator.wikimedia.org/T179229#4054176 (10Pchelolo) [18:34:35] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.005 second response time [18:37:05] 10Release-Engineering-Team (Kanban), 10Discovery, 10Wikidata, 10Wikidata-Query-Service, 10User-Smalyshev: Set up user for automatic WDQS GUI builds - https://phabricator.wikimedia.org/T189811#4054206 (10Smalyshev) p:05Triage>03Normal [18:37:23] 10Continuous-Integration-Infrastructure, 10Front-end-Standards-Group, 10MediaWiki-extensions-General, 10Services (designing): Decide whether we want the package-lock.json to commit or ignore - https://phabricator.wikimedia.org/T179229#4054220 (10Pchelolo) We've recently filed T189805 to discuss the same qu... [18:45:20] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#4054235 (10awight) [18:51:40] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054249 (10awight) @mmodell I'm running into some scap weirdness around LFS, just wanted to keep you posted. The setup is currently that submodule ores/assets has git-lfs enabled, and the test.bin.gz file is stored in git-... [18:54:29] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:54:56] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#4054251 (10MBinder_WMF) Request to add @LGoto so that she can manage boards for her new teams working on #wikipedia-android-app-backlog and #wikipedia-ios-app-backlog and their associat... [19:00:06] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054259 (10awight) Reading through the scap code, it seems it won't do the right thing. Since we have LFS enabled in a submodule, it looks like we need to go through each submodule and run "git lfs pull". This is horrible... [19:00:20] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054260 (10mmodell) @awight: That's odd. With `git_binary_manager: 'git-lfs'` set, you should see scap logging the following, at the end of the fetch stage: Pulling large objects [using git-lfs] This message is `info`... [19:03:00] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054268 (10mmodell) @awight: no lfs isn't copied to the deploy host, and phabricator doesn't have any place to store lfs files yet (working on that) so large files must be fetched directly from the origin. I'll see what I c... [19:03:53] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054269 (10Paladox) @mmodell, he was using gerrit to pull in the lfs (as lfs is enabled on that repo in gerrit) :) [19:04:51] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054270 (10mmodell) @paladox: I understand that much. But scap rewrites the git origins, it needs to //not// rewrite the lfs origin. [19:11:48] 10Phabricator: Setting up phabricator for the first results in schema errors - https://phabricator.wikimedia.org/T189601#4054277 (10mmodell) Temporary workaround: Run `sudo /srv/phab/phabricator/bin/storage shell` and paste the following sql statements: ```lang=sql ALTER TABLE phabricator_maniphest.maniphest_t... [19:11:58] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054278 (10Paladox) ah ok, what if we did git lfs pull before we scap deploy? [19:13:34] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054281 (10mmodell) @paladox: see T180627#4054259, that doesn't quite work because the lfs origin got rewritten to point at tin instead of gerrit. [19:14:26] 10Phabricator, 10Release-Engineering-Team (Kanban): Setting up phabricator for the first results in schema errors - https://phabricator.wikimedia.org/T189601#4054282 (10mmodell) a:03mmodell [19:14:44] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054284 (10Paladox) @mmodell yep, though the file would be on tin then similar to git-lfs :). So currently the file has lfs text, when you pull it, it becomes the normal file as hosted in lfs storage. [19:14:59] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054285 (10Paladox) oh i see. [19:31:01] what is the right channel now where i see the alerts for broken puppet runs on labs [19:33:41] right, cloud-feed [19:40:52] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<55.56%) [19:52:30] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [10.0] [20:09:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Wikidata: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4054374 (10Smalyshev) p:05Triage>03High [20:30:05] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #147: 04FAILURE in 43 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/147/ [20:44:49] 10Scap, 10Operations, 10Packaging, 10Patch-For-Review: Install git-lfs client (at least on scap targets & masters) - https://phabricator.wikimedia.org/T180628#4054495 (10mmodell) >>! In T180628#4050430, @demon wrote: > I can't think of any compelling reason why it would be *required* on the masters...but I... [20:54:23] (03PS2) 10Hashar: Port mw-install-sqlite and mw-install-mysql to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 [20:54:46] (03CR) 10jerkins-bot: [V: 04-1] Port mw-install-sqlite and mw-install-mysql to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 (owner: 10Hashar) [20:55:33] thcipriani: I got quibble to run PHPUnit tests again :D [20:55:42] nice :) [20:55:43] I wasted my time debugging a few failures [20:55:50] only to find out the tests fail on sqlite [20:55:54] but they mostly pass using mysql ! [20:56:11] https://gerrit.wikimedia.org/r/#/c/419787/2/quibble/mediawiki/maintenance.py spawns mysql for me :] [20:58:21] seems like that would work [20:58:23] :) [20:58:29] 10MediaWiki-Codesniffer: UnusedUsed sniff did not catch https://gerrit.wikimedia.org/r/#/c/418023/ - https://phabricator.wikimedia.org/T189323#4054553 (10Umherirrender) p:05Triage>03Normal a:03Umherirrender [20:58:44] * thcipriani tries to find local quibble install [21:01:20] (03PS1) 10Legoktm: Forbid usage of assert() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419854 [21:01:35] Error: 1071 Specified key was too long; max key length is 767 bytes (localhost:/tmp/mysqld/mysqld.sock)
[21:01:36] bah [21:02:26] (03CR) 10MaxSem: [C: 032] "DIEDIEDIE" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419854 (owner: 10Legoktm) [21:03:17] (03Merged) 10jenkins-bot: Forbid usage of assert() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419854 (owner: 10Legoktm) [21:03:44] (03CR) 10jenkins-bot: Forbid usage of assert() [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419854 (owner: 10Legoktm) [21:04:49] (03CR) 10Hashar: Forbid usage of assert() (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/419854 (owner: 10Legoktm) [21:04:58] legoktm: ahhh you merged too fast :] [21:05:09] legoktm: https://gerrit.wikimedia.org/r/#/c/419854/1//COMMIT_MSG@10 :] [21:05:17] whoops [21:05:25] no worries [21:05:29] it is just the commit message [21:05:39] :) [21:06:06] thcipriani: oh and the suite fails when LANG != C.UTF-8 :] [21:06:08] magic finding [21:08:23] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:13:09] twentyafterfour: Do you happen to know—is git-lfs supported by both gerrit and Phabricator? I was surprised that my "observing" repo in Phabricator seemed to be mirroring the LFS changes correctly. [21:13:42] awight lfs replaces the content in the file [21:13:44] with lfs content [21:13:52] so what your seeing is not the actual content [21:14:07] on gerrit lfs objects are stored in a seperate repo folder. [21:14:16] so git lfs pull will pull those objects [21:14:39] So if I clone this repo from Diffusion, I'll still be lfs pulling from gerrit? [21:15:43] nope, if you clone it from diffusion [21:15:56] you wont be able to pull the lfs objects from gerrit [21:16:24] That's the part that impressed me. I committed to gerrit, then cloned from the Phabricator mirror, and the LFS files were correct. [21:16:58] awight really? [21:17:06] awight link please :) [21:17:09] right??? It should have failed [21:18:10] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054667 (10mmodell) @awight: I think there is an easy solution. Really more of a workaround, until scap learns to do this automatically: In each submodule, configure lfs.url to point to gerrit: git config -f .lfsconfi... [21:18:32] paladox: https://phabricator.wikimedia.org/source/ores-assets.git [21:18:44] thanks [21:19:13] PROBLEM - SSH on integration-slave-docker-1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:45] innodb_large_prefix = 1 [21:19:46] pff [21:20:29] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054670 (10awight) @mmodell Thanks, I'm okay with workarounds as long as they're sorta sustainable. It sounds like I can do these steps myself, other than the last part about how the clone is done. I'll try that out on th... [21:20:42] awight https://phabricator.wikimedia.org/R2300:4aca5726b6571c904a9103e59c092324edd2ad06 ? [21:20:51] the file has valid lfs data [21:20:56] when you git lfs pull [21:20:59] \p/ hehe [21:21:02] that will mean you will have the file [21:21:07] No idea why it works, but cool. [21:21:09] awight that data is expected [21:21:11] though [21:21:28] it just replaced the file contents [21:21:30] with lfs blobs [21:21:42] so when you git lfs pull, you gain the full file (and the full size back) [21:22:58] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054677 (10mmodell) @awight: cool, ping me here or on irc if you run into more trouble. I'll be happy to work through it with you. In the meantime I'm going to work on a scap patch to address `git lfs clone`. [21:23:30] twentyafterfour: Already in trouble :-) batch response: Repository or object not found: https://gerrit.wikimedia.org/repo/lfs/url/objects/batch [21:23:44] awight lol [21:23:49] awight that's the wrong url [21:24:02] awight you were ment to replace that url :) [21:24:04] RECOVERY - SSH on integration-slave-docker-1001 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [21:24:09] (he used that one as an example) [21:24:20] * awight scowls [21:26:26] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054701 (10mmodell) My apologies. I meant for 'repo/lfs/url' to be a //**variable**//... phabricator didn't like my formatting so I submitted it without proper emphasis. I think it should be set to the gerrit https clone... [21:26:47] awight: yeah my bad, it was supposed to be a variable [21:27:01] phabricator didn't format it correctly and I got lazy when I submitted the example [21:27:33] Time: 2.7 minutes, Memory: 902.53MB [21:27:33] OK, but incomplete, skipped, or risky tests! [21:27:33] Tests: 17605, Assertions: 108719, Skipped: 27, Risky: 5. [21:27:37] thcipriani: quibble works :] [21:28:05] {{done}} [21:28:08] (03PS3) 10Hashar: Port mw-install-sqlite and mw-install-mysql to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 [21:28:23] well solely with mediawiki/core / vendor / mysql [21:28:41] (03CR) 10jerkins-bot: [V: 04-1] Port mw-install-sqlite and mw-install-mysql to python [integration/quibble] - 10https://gerrit.wikimedia.org/r/419787 (owner: 10Hashar) [21:32:17] twentyafterfour: The forecast is hazy. I found this documentation, https://gerrit-review.googlesource.com/c/plugins/lfs/+/80739/1/src/main/resources/Documentation/config.md [21:32:28] I think they might be serious about http vs https? [21:33:21] well, either way, this fails: https://gerrit.wikimedia.org/scoring/ores/assets/info/lfs [21:33:40] https://gerrit.googlesource.com/plugins/lfs/+/master/src/main/resources/Documentation/ [21:38:29] twentyafterfour: I got close with this URL, https://gerrit.wikimedia.org/r/scoring/ores/assets/info/lfs, but then I'm asked for a username and password for gerrit. [21:40:42] awight yeh [21:42:20] i created https://bugs.chromium.org/p/gerrit/issues/detail?id=8555 [21:43:13] cool, thanks [21:44:13] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054754 (10mmodell) why running `git review` ? Just attempting to commit the lfsconfig to gerrit? @demon: any ideas? [21:46:11] awight does pulling from https://gerrit.wikimedia.org/r/scoring/ores/assets/info/lfs, work without a username or password? [21:46:17] twentyafterfour: Yeah it's strange. On the bright side, either the git client or server was sanity-checking the LFS URL [21:47:13] twentyafterfour: another possibility: I could set the lfs.url to point to Diffusion, would that be an acceptable request to make from deployment targets? [21:47:41] awight: phabricator's git lfs support is not ready yet [21:47:48] kk [21:48:07] hmm. I need to change the submodule pointer, then... [21:48:08] I mean, phabricator supports it but I need a storage back-end to store the files, currently that is the mysql database which isn't good [21:48:37] awight: submodule is pointed to phabricator? [21:48:43] yes [21:48:59] that may be the problem then? hmm [21:49:04] I was... shooting for consistency with the other submodules, but that was silly. [21:49:13] kk fixing and trying again. [21:49:37] I'm working on phab storage, should have it soon [21:49:46] e.g. in a few weeks I hope [21:50:07] :) [22:00:47] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054807 (10awight) Meanwhile, I fixed the submodule pointer to pull submodules/assets from gerrit. Now the deployment silently fails. Running some of the steps manually, ``` sudo su - deploy-service git submodule sync --r... [22:02:34] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054820 (10mmodell) and you ran `git lfs install --global` as the deploy user? Another thing to try is `git lfs install --local` inside the repo on the target. [22:06:16] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054833 (10awight) The "--global" command doesn't seem to work. I ran "git lfs install" in the submodule repo, and get the same results. Here's the latest, ``` deploy-service@deployment-ores01:/srv/deployment/ores/deploy/... [22:07:36] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054837 (10mmodell) @swight: OK I'm going to poke at it with a stick, let me see if I can figure out what's up. [22:16:04] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4054843 (10JAufrecht) For an experiment, I tweaked the load so that it parsed all of the "bad" transactions, treating the list of PHIDs as a... [22:17:25] looks like a new lts release was done of jenkins https://jenkins.io/changelog-stable/ [22:25:36] PROBLEM - Free space - all mounts on integration-slave-docker-1005 is CRITICAL: CRITICAL: integration.integration-slave-docker-1005.diskspace.root.byte_percentfree (<40.00%) [22:27:46] legoktm integration-slave-docker-1005 is out of space ^^ [22:31:53] 10Phabricator, 10Wikimedia-Hackathon-2018: Build a bot that pushes Phabricator updates to Google Chat - https://phabricator.wikimedia.org/T189313#4038677 (10Joe) >>! In T189313#4047905, @Framawiki wrote: > Just a note : according to https://wikitech.wikimedia.org/wiki/Mail, lots of @wikimedia.org mail address... [22:33:29] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T183964#4054902 (10Paladox) [22:34:44] awight: ok lfs works in the submodule after setting lfs.url like this: [22:34:46] git config lfs.url 'https://gerrit.wikimedia.org/r/scoring/ores/assets/info/lfs' [22:35:20] (03PS1) 10Jdlrobson: Use `npm run docs` not jsduck [integration/config] - 10https://gerrit.wikimedia.org/r/419941 (https://phabricator.wikimedia.org/T138401) [22:35:27] twentyafterfour yay [22:35:31] so it works anon? [22:35:42] (no need to set a password or username?) [22:36:09] paladox: right [22:36:19] :) [22:38:32] 10Scap, 10Patch-For-Review: Support git-lfs - https://phabricator.wikimedia.org/T180627#4054928 (10mmodell) This seems to have worked, finally: ```lang=bash sudo -u deploy-service bash cd /srv/deployment/ores/deploy-cache/revs/d54a2577626b7164404ea8135d8a5cb5f4c00da8/submodules/assets git config lfs.url 'https... [22:40:20] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T183964#4054949 (10Reedy) [22:41:46] oh well https://github.com/google/gitiles/issues/129#issuecomment-373540645 that will make some one happy heh [23:09:20] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T183964#4055100 (10Daimona) [23:35:03] 10Phabricator, 10Wikimedia-Hackathon-2018: Build a bot that pushes Phabricator updates to Google Chat - https://phabricator.wikimedia.org/T189313#4038677 (10Tgr) >>! In T189313#4046664, @Aklapper wrote: > Seeing how quickly proprietary projects can decide to disable team productivity features (like [[ https://... [23:58:08] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Phlogiston: Phlogiston reports don't have new data since mid-February - https://phabricator.wikimedia.org/T188149#4055249 (10mmodell) @JAufrecht: ah ha! Thanks to your detailed comparison, I figured out what changed: https://phabricator.wikimedia.org/T188...