[00:21:24] Krenair: what host in beta do I use to run maint scripts? [00:21:43] -tin will do [00:21:58] don't think we have a separate host for it like in prod [00:23:21] hmm [00:23:31] legoktm@deployment-tin:~$ mwscript ../../../../../home/legoktm/parsoidAllPages.php --wiki=enwiki [00:23:31] The MediaWiki script file "/srv/mediawiki-staging/php-master/../../../../../home/legoktm/parsoidAllPages.php" does not exist. [00:23:43] legoktm@deployment-tin:~$ less /srv/mediawiki-staging/php-master/../../../../../home/legoktm/parsoidAllPages.php <-- works though [00:27:37] legoktm: Just drop it in WikimediaMaintenance :p [00:27:41] Or maintenance/ [00:27:45] If it's just a tmp thing heh [00:28:23] :D woot working now [00:31:29] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859024 (10Paladox) Some answers here https://groups.google.com/forum/#!topic/repo-discuss/SesSyGPbNoI Seems they are mixing old code and new code. [00:44:21] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Run etsy/phan from jenkins for MediaWiki core - https://phabricator.wikimedia.org/T132636#2859045 (10Legoktm) Okay awesome! I'll look into hooking it jenkins with checkstyle and moving it back to nodepool. [01:23:24] Hmm. Did we drop git versions from Beta Cluster intentionally or by accident? [01:23:56] https://en.wikipedia.beta.wmflabs.org/w/api.php?action=query&meta=siteinfo&format=jsonfm&siprop=extensions "vcs-version"s are no longer populated, as used by VE; also (not) showing up in Special:Version. [01:38:25] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2859090 (10Pchelolo) I've made a PR to make the process simpler: https://github... [01:44:02] James_F: Not intentionally. There's something wonky going on with that in scap right now [01:44:04] On my radar. [01:44:29] * James_F nods. [01:44:44] The cache_git_info stuff is a little weird. Tbh, we're surprised it actually ever worked to begin with ;-) [01:44:53] Yeah. [03:20:59] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [03:30:58] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [03:42:52] (03PS1) 10Legoktm: phan: Output a checkstyle report and link to the result [integration/config] - 10https://gerrit.wikimedia.org/r/326069 [03:43:17] (03CR) 10Legoktm: [V: 032 C: 032] phan: Output a checkstyle report and link to the result [integration/config] - 10https://gerrit.wikimedia.org/r/326069 (owner: 10Legoktm) [03:43:28] oops, didn't mean to press V2 [03:44:46] (03Merged) 10jenkins-bot: phan: Output a checkstyle report and link to the result [integration/config] - 10https://gerrit.wikimedia.org/r/326069 (owner: 10Legoktm) [03:45:05] !log deploying https://gerrit.wikimedia.org/r/326069 [03:45:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:57:35] (03PS1) 10Legoktm: phan: Move back to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/326070 [03:59:07] (03CR) 10Legoktm: [C: 032] phan: Move back to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/326070 (owner: 10Legoktm) [04:00:42] (03Merged) 10jenkins-bot: phan: Move back to nodepool [integration/config] - 10https://gerrit.wikimedia.org/r/326070 (owner: 10Legoktm) [04:01:02] !log deploying https://gerrit.wikimedia.org/r/326070 [04:01:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:05:36] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #228: 04FAILURE in 9 min 36 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/228/ [05:14:52] 06Release-Engineering-Team, 06Services, 10Phabricator (Upstream), 07Regression, 07Upstream: Phabricator upgrade broke milestone tag completion - https://phabricator.wikimedia.org/T152726#2859255 (10mmodell) Note that searching for the milestone name (without the parent project) seems to find them reliabl... [05:17:02] 06Release-Engineering-Team, 06Services, 10Phabricator (Upstream), 07Regression, 07Upstream: Phabricator upgrade broke milestone tag completion - https://phabricator.wikimedia.org/T152726#2859256 (10mmodell) Seems like the fuzzy filter doesn't like '(' in the search terms but also doesn't tolerate more th... [06:43:42] Yippee, build fixed! [06:43:42] Project selenium-Wikibase » chrome,test,Linux,contintLabsSlave && UbuntuTrusty build #202: 09FIXED in 2 hr 3 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/202/ [08:36:32] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859392 (10Paladox) We have a patch https://gerrit-review.googlesource.com/#/c/92830/ that I want to try out on the test install [08:41:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2859397 (10hashar) The Jenkins job only does `npm install && npm test`. Either... [08:47:33] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859403 (10Niharika) >>! In T152640#2857176, @Paladox wrote: > @Niharika hi could you try Niharika29 please? That worked for me. Thanks @Paladox! :) [08:55:26] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859406 (10Paladox) Your welcome [09:43:38] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10jcrespo) It is now affecting jynus -> Jcrespo. "Cannot assign user name "jynus" to account 4318; name already in use.". [09:51:43] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859470 (10jcrespo) It turns out I have to log in with lowercase, which is not my LDAP name. Now it works, but I have to use lower case for gerrit and uppe... [10:40:40] paladox: thanks for the Gerrit repo-discuss email and the https://bugs.chromium.org/p/gerrit/issues/detail?id=5090 :)} [10:41:43] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859510 (10hashar) [10:42:02] oh that got filled by Chad [10:42:03] bah [10:44:22] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859514 (10hashar) The investigation is being conducted with upstream authors on https://bugs.chromium.org/p/gerrit/issues/detail?id=5090 [10:57:10] 10Continuous-Integration-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 05WMF-deploy-2016-12-13_(1.29.0-wmf.6): Run etsy/phan from jenkins for MediaWiki core - https://phabricator.wikimedia.org/T132636#2859520 (10hashar) https://gerrit.wikimedia.org/r/#/c/326043/ now let us pass extra argumen... [11:07:50] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2859543 (10Paladox) Also the discussions is happening here about that too https://groups.google.com/forum/#!topic/repo-discuss/SesSyGPbNoI We may have to... [11:10:40] 03Scap3, 10scap: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#2859546 (10hashar) [11:13:17] 03Scap3, 10scap: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#2859548 (10hashar) a:03thcipriani Tyler has a patch for it D503 :} [11:52:25] 03Scap3 (Scap3-MediaWiki-MVP), 10Electron-PDFs, 03WMDE-QWERTY-Team-Board: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2859565 (10Addshore) It looks like after one of the scaps for the train the message issue has reslved itself. [12:11:30] 06Release-Engineering-Team, 06Services, 10Phabricator (Upstream), 07Regression, 07Upstream: Phabricator upgrade broke milestone tag completion - https://phabricator.wikimedia.org/T152726#2859613 (10mobrovac) In my exp, typing `done` has never turned out `Services (done)` [13:14:02] 06Release-Engineering-Team, 06Services, 10Phabricator (Upstream), 07Regression, 07Upstream: Phabricator upgrade broke milestone tag completion - https://phabricator.wikimedia.org/T152726#2859641 (10epriestley) Although there's no way to know this without knowledge of the internals, typeaheads are (almost... [13:29:49] 06Release-Engineering-Team, 10MediaWiki-API, 10Monitoring, 06Operations, and 2 others: API action=parsoid-batch not available on Graphite - https://phabricator.wikimedia.org/T152776#2859704 (10hashar) [13:38:38] 06Release-Engineering-Team, 06Services, 10Phabricator (Upstream), 07Regression, 07Upstream: Phabricator upgrade broke milestone tag completion - https://phabricator.wikimedia.org/T152726#2859724 (10epriestley) I think this is now (well, shortly) fixed upstream, likely available here after the next upgrad... [13:46:00] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #238: 04FAILURE in 2 min 0 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/238/ [13:56:12] 06Release-Engineering-Team, 06Operations, 06Parsing-Team, 07HHVM, and 3 others: API cluster failure / OOM - https://phabricator.wikimedia.org/T151702#2828786 (10Vriullop) I try to expand it from the wiki part. A page with this template transclusion was rendered in 5-8 seconds according with parser profile... [14:14:11] 06Release-Engineering-Team, 06Operations, 06Parsing-Team, 07HHVM, and 3 others: API cluster failure / OOM - https://phabricator.wikimedia.org/T151702#2859773 (10mobrovac) >>! In T151702#2859748, @Vriullop wrote: > I try to expand it from the wiki part. A page with this template transclusion was rendered in... [14:32:36] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10Mglaser) Hello everyone, my account is also affected. Gerrit tells me "Cannot assign user name "mglaser" to account 4327; name already in use." [14:40:34] 06Release-Engineering-Team, 10MediaWiki-API, 10Monitoring, 06Operations, and 3 others: API action=parsoid-batch not available on Graphite - https://phabricator.wikimedia.org/T152776#2859816 (10Anomie) [15:12:11] 06Release-Engineering-Team, 06Operations, 10Wikimedia-Logstash, 06Services (watching): Kibana functionality missing after upgrade: histograms - https://phabricator.wikimedia.org/T152782#2859881 (10GWicke) [15:16:18] 06Release-Engineering-Team, 06Operations, 10Wikimedia-Logstash, 06Services (watching): Kibana functionality missing after upgrade: histograms - https://phabricator.wikimedia.org/T152782#2859908 (10mobrovac) [15:25:25] 06Release-Engineering-Team, 06Operations, 10Wikimedia-Logstash, 06Services (watching): Kibana functionality missing after upgrade: histograms - https://phabricator.wikimedia.org/T152782#2859881 (10EBernhardson) Poking through the 'Visualize' tab, kibana 4 reports having both standard and date based histogr... [15:32:49] 06Release-Engineering-Team, 06Operations, 10Wikimedia-Logstash, 06Services (watching): Kibana functionality missing after upgrade: histograms - https://phabricator.wikimedia.org/T152782#2859969 (10GWicke) The workflow we were using is more along the lines of: 1) Query some subset of log entries, typically... [15:39:47] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:51:05] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:14:18] hashar your welcome :), i am testing some fixes they did as i can reproduce it on local dev install but doint want to try logging in on prod with Paladox as i doint want to break my account just in case. [16:14:29] and i just saw your message :) [16:15:20] though i am helping Chad with the debug since we are using gerrit.git.wmflabs.org as a test for the fixes as it runs the puppet class that prod gerrit uses. [16:16:03] He did file that issue upstream though :) [16:16:17] he = chad. [16:17:39] (03PS1) 10Hashar: Make .gitreview to point to debian branch [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/326140 [16:18:05] (03CR) 10jenkins-bot: [V: 04-1] Make .gitreview to point to debian branch [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/326140 (owner: 10Hashar) [16:21:42] (03CR) 10Hashar: "And of course changing the .gitreview in the debian branch, clash with the "upstream" one has a different defaultbranch parameter:" [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/326140 (owner: 10Hashar) [16:30:38] (03PS2) 10Hashar: Make .gitreview to point to debian branch [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/326140 [16:30:59] (03CR) 10jenkins-bot: [V: 04-1] Make .gitreview to point to debian branch [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/326140 (owner: 10Hashar) [16:32:16] (03CR) 10Hashar: "Not really :(" [integration/uprightdiff] (debian) - 10https://gerrit.wikimedia.org/r/326140 (owner: 10Hashar) [16:33:22] * hashar gives up [16:33:44] paladox: yeah quite helpful to have some kind of dev platform [16:33:47] thx for all of that! [16:34:02] Yep your welcome :). [16:34:13] hashar i think we may have a workaround using lower case config [16:34:14] for ldap [16:34:26] which i found allowed me to login using paladox and Paladox [16:34:39] but it broke users accounts who used a capital in there username like Chad. [16:36:58] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860041 (10Paladox) [16:37:23] hashar i just added an important block to ^^ to show our workarounds. [16:39:22] paladox: So caught up with repo-discuss this morning. It looks like we'll need to do 2 things for sure (maybe 3). [16:39:28] (1) Downtime to fix/reindex the broken rows [16:39:32] Yep [16:39:43] i have applied there fix for the external id deleting [16:39:44] Well, 1 thing for sure ^ [16:40:00] Maybe (2): Pull in the stable-2.13 patch to avoid the row deletions (https://gerrit-review.googlesource.com/#/c/92832/) [16:40:06] ostriches could you try logging in with demon on gerrit.git.wmflabs.org just wondering if that will work. [16:40:10] Maybe (3): Convert to all lowercase (maybe, I'm undecided) [16:40:13] Yeh [16:40:56] I stil get the error, but we haven't fixed account_external_ids and reindexed yet. [16:40:57] ostriches https://gerrit-review.googlesource.com/#/c/92833/ [16:41:03] ok [16:42:00] How the upstream discussion went: "There's no way rows are being deleted" -> "I promise they are" -> "Oh, I see how that could happen" -> "Here's a fix!" [16:42:02] hehe [16:42:08] LOL [16:42:17] \O/ [16:42:32] ostriches does logging in with chad work?, ive removed the config that converted to lower case as i want your user to have two tables so i can test there fix. [16:42:58] Nope, neither works. I only have 1 entry right? That needs fixing + reindexing. The fix just keeps it from happening *again* [16:43:04] It doesn't unbreak a broken account [16:43:05] Oh [16:43:18] yep [16:43:26] I'm also tempted to pull in https://gerrit-review.googlesource.com/#/c/92830/ to our stable-2.13 [16:43:33] It won't land in master tho :\ [16:43:57] I pulled that too [16:44:38] that dosent fix logging in (same captial problems) [16:44:42] I mean yeah we can easily pull, I'm just referring to Edwin's comment: "As a workaround this is good, but we shouldn't merge it. We should rather understand what is the actually problem." [16:44:50] The capital issue doesn't bug me as much, long as people *can* login [16:44:51] yep. [16:45:07] apparently oauth dosent have this problem [16:45:19] according to upstream so it's something in there schema for ldap that is [16:51:24] (https://github.com/gerrit-review/gerrit/commit/b06cfdbe4b18bcaedaffbee88d4d158f1b254572) [16:55:24] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860060 (10demon) >>! In T152640#2859470, @jcrespo wrote: > It turns out I have to log in with lowercase, which is not my LDAP name. Now it works, but I ha... [17:07:53] ostriches the convert to all lowercase will fix logging in no matter what caps your username is in as long as you created it in lowercase. What i doint know is if it will cause other effects and will it work converting all users to lowercase if we use that tool that should do it. [17:08:25] It should be fine to convert. In a meeting right now. [17:08:37] meeting && end of week for me! [17:08:40] see you next week [17:08:44] ok [17:34:53] https://github.com/gerrit-review/gerrit/commit/b3b0d294673ed5c41677583dbb6e5de22db65bb9 [17:59:59] ostriches the fix to external id seems to work [18:00:20] i managed to add gerrit:Chad to the db to allow you to login, i then did select * from account_external_ids where account_id = 4; [18:00:23] woops [18:00:36] update account_external_ids set external_id = 'gerrit:Chad' where external_id = 'gerrit:demon'; [18:00:55] I was following https://cyborthodox.wordpress.com/2015/02/08/gerrit-authentication-openid-to-ldap/ [18:02:36] im running java -jar gerrit.war reindex -d review_site --threads 4 now [18:04:12] I switched the config back on [18:04:14] for lower case [18:10:24] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2860190 (10Pchelolo) >>! In T152684#2859397, @hashar wrote: > The Jenkins job o... [18:15:53] ostriches Hi could you try logging in using Chad and chad on gerrit.git.wmflabs.org please? [18:16:09] i fixed the table so you have a gerrit:chad in the row now. [18:17:00] Upper and lowercase work [18:17:50] Yay [18:18:05] ostriches i found a fix for the captial problems [18:18:07] then [18:18:11] well actually you did [18:18:29] I'm just glad I can login & not lose my account_external_ids row :p [18:19:00] Yep [18:19:06] i did not see any rows go either [18:19:14] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap [18:19:24] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#ldap.localUsernameToLowerCase [18:19:39] ^^ that config is to allow you to do for example Paladox paladox [18:20:04] the config is confusing as the name suggests that it will not allow you to use upper case. [18:22:51] 10Gerrit, 06Release-Engineering-Team, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860226 (10Paladox) I've just tested some upstream changes which fixed dropping of the rows in the external id table. Also @demon has found a fix for logg... [18:59:23] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2860349 (10Nuria) And (call me crazy) couldn't we make jenkins spawn docker con... [19:29:18] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:33:25] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860446 (10Paladox) Actually case insensitive now works with a reindex @demon tried on my account. [19:34:06] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2860447 (10greg) >>! In T152684#2860349, @Nuria wrote: > And (call me crazy) co... [19:35:45] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2855839 (10Jdforrester-WMF) Yesterday I ran into this and worked around it by logging in as `Jforrester`, but now this doesn't work a... [19:36:28] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Reading-Web-Trending-Service, and 2 others: Setup change-propagation service CI - https://phabricator.wikimedia.org/T152684#2860452 (10GWicke) @Nuria, we discussed using containers for CI several times b... [19:38:43] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860457 (10Paladox) @Jdforrester-WMF hi, @demon just applied some bugfixes which will break login for some users. We need to fix the... [19:39:11] yo, gwicke ! neat ^ So, we had a fruitful discussion with ops yestrerday about that topic. I'm going to add you to a gdoc that we used as our reference so you can see what we talked about. [19:39:27] re containers [19:39:30] not gerrit :) [19:42:44] (shared) [19:46:05] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860464 (10demon) Ok, so here's where we're at: * We just applied some fixes that should keep this from **expanding** to more users a... [19:49:35] 06Release-Engineering-Team, 15User-greg: Setup monthly "QA Tribe" meeting - https://phabricator.wikimedia.org/T152570#2852845 (10greg) 05Open>03Resolved [19:52:28] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860486 (10demon) Ok, so all users should be fixed and able to login again. The case sensitivity is still a little tight so please be... [20:02:46] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860512 (10Papaul) Thanks @demon I am able to login. [20:09:16] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:09:41] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860524 (10Huji) Worked for me too. [20:10:37] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860527 (10Huji) [20:17:04] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review, 07Regression, 07Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#2860531 (10demon) p:05Unbreak!>03High Lowering priority a tad. Underlying breakage is fixed now and shouldn't resurface, but we'l... [20:50:52] Nikerabbit: still around? https://phabricator.wikimedia.org/T152800 looks pretty bad. [20:57:28] 10Continuous-Integration-Infrastructure, 05MW-1.29-release-notes, 13Patch-For-Review, 05WMF-deploy-2016-12-13_(1.29.0-wmf.6): Run etsy/phan from jenkins for MediaWiki core - https://phabricator.wikimedia.org/T132636#2860601 (10Legoktm) Yep, sorry I set up the checkstyle stuff last night, and forgot to comm... [21:00:54] 10Gerrit, 06Release-Engineering-Team: Support redis as a cache store - https://phabricator.wikimedia.org/T152802#2860605 (10demon) [21:01:00] Just a note if people run into it – there's a new version of eslint released a few hours ago, and grunt-eslint doesn't pin versions so if you get sudden new behaviours with masters not passing ping me. I encountered one issue in VE which we worked around (and reported upstream) but there may be more. [21:04:33] greg-g: Hey. Just a heads-up about https://phabricator.wikimedia.org/T152800 - I've set that to UBN (and pinged Niklas above who might be out already) [21:24:45] 06Release-Engineering-Team, 10MediaWiki-API, 10Monitoring, 06Operations, and 3 others: API action=parsoid-batch not available on Graphite - https://phabricator.wikimedia.org/T152776#2859704 (10Legoktm) I don't think this is going to be that useful....the numbers will be all over the place, we really just n... [21:26:29] andre__: RoanKattouw wrote a quick fix which I'm merging. [21:26:41] James_F: Thank you! [21:26:53] If greg-g or ostriches say yes RoanKattouw'll deploy it. [21:30:43] James_F: godspeed [21:30:57] RoanKattouw: ^^ [21:34:53] In a meeting, deploying in 30 [21:50:19] 10Gerrit, 10MediaWiki-Vagrant: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#2860709 (10Tgr) This is the manual output: ``` $ /usr/bin/git clone --recurse-submodules https://gerrit.wikimedia.org/r/p/mediawiki/core.git /vagrant/mediawiki Cloning into... [22:30:22] 10Gerrit, 10MediaWiki-Vagrant: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#2860792 (10Tgr) With more logging: ``` $ GIT_CURL_VERBOSE=1 GIT_TRACE=1 GIT_TRACE_PACK_ACCESS=1 GIT_TRACE_PACKET=1 GIT_TRACE_PACKFILE=1 /usr/bin/git clone -v -v --recurse-sub... [22:42:23] 10Gerrit, 10MediaWiki-Vagrant: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#2860821 (10Tgr) Depending on whether the verbose logging counts started or finished packets, the offending one is either `b7e3ddf438d457d02d54da232ad7c7ba1a2d87940032` (`refs... [23:02:26] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:05:23] 10Gerrit, 10MediaWiki-Vagrant: "index-pack failed" when installing new MediaWiki-Vagrant box - https://phabricator.wikimedia.org/T152801#2860882 (10Tgr) Worked around by cloning via SSH and then editing the remote URL back to HTTPS. [23:08:04] 03Scap3 (Scap3-MediaWiki-MVP), 10Electron-PDFs, 03WMDE-QWERTY-Team-Board: ElectronPdfService mw extension l10n messages missing after full scap sync - https://phabricator.wikimedia.org/T152424#2847642 (10greg) Any other debugging we can do now? :/