[00:03:17] Krenair it would be puppet [00:03:23] puppet installs the update [00:03:26] ... [00:03:41] https://gerrit.wikimedia.org/r/c/operations/puppet/+/438121 [00:04:04] yeah [00:05:29] gonna cherry-pick that, thanks paladox [00:05:37] ok your welcome :) [00:05:48] im not sure if that [00:05:54] has been uploaded to wikimedia apt [00:05:56] yet Krenair [00:06:12] doesn't matter for deployment-prep [00:06:20] the version that begins to use is clearly in deployment-prep apt [00:06:29] ok [00:08:30] oh wait [00:08:49] Error: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists... [00:09:03] it's not actually telling it to install the version in puppet [00:10:46] except puppet knows what it's supposed to be doing: [00:10:48] Error: /Stage[main]/Scap/Package[scap]/ensure: change from 3.9.0-1+0~20180515171834.349~1.gbpcef62d to 3.8.2-1+0~20180607230422.353~1.gbp2bb4cc failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install scap' returned 100: Reading package lists... [00:11:33] is puppet supposed to be detecting that as a downgrade and passing --force-yes ? [00:12:05] puppet cannot tell if it's a downgrade i think. [00:12:15] apt is returning that it's a downgrade though [00:12:52] puppet clearly knows what versions it's dealing with [00:13:11] /ensure: change from 3.9.0-1+0~20180515171834.349~1.gbpcef62d to 3.8.2-1+0~20180607230422.353~1.gbp2bb4cc [00:13:40] apt will definitely complain about that [00:28:38] 10MediaWiki-Releasing, 10MW-1.27-release, 10MW-1.27-release-notes, 10PHP 7.1 support: Make MediaWiki 1.27 (LTS) compatible with PHP 7.1 - https://phabricator.wikimedia.org/T174262#4269641 (10Krinkle) [00:42:47] !log cumin 'P{R:Package = scap} and P{F:lsbdistcodename = jessie}' 'apt-get install scap=3.8.2-1+0~20180607230422.353~1.gbp2bb4cc -y --force-yes' [00:42:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:53:44] 10Beta-Cluster-Infrastructure: Puppet failure on deployment-cpjobqueue - https://phabricator.wikimedia.org/T196829#4269648 (10Krenair) [00:55:34] 10Beta-Cluster-Infrastructure: Puppet failure on deployment-cassandra3-02 - https://phabricator.wikimedia.org/T196830#4269659 (10Krenair) [00:56:07] 10Beta-Cluster-Infrastructure: Puppet failure on deployment-cassandra3-0[12] - https://phabricator.wikimedia.org/T196830#4269670 (10Krenair) [00:56:15] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [00:57:11] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [00:57:23] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [00:58:21] 10Beta-Cluster-Infrastructure, 10Services: Puppet failure on deployment-cpjobqueue - https://phabricator.wikimedia.org/T196829#4269671 (10Krenair) Host created by @mobrovac [00:58:43] 10Beta-Cluster-Infrastructure, 10Services: Puppet failure on deployment-cassandra3-0[12] - https://phabricator.wikimedia.org/T196830#4269674 (10Krenair) Hosts created by @mobrovac [00:59:30] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:59:42] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:12] RECOVERY - Puppet errors on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:14] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:00:23] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:01] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:01:11] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:00] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:07] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:11] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:19] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:41] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:56] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [01:04:08] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [03:57:02] 10Beta-Cluster-Infrastructure, 10Operations: Mails through deployment-mx SPF & DKIM fails - https://phabricator.wikimedia.org/T87338#4269749 (10Krenair) Probably, I should probably add an SPF record allowing this host to send mail [04:08:34] 10Beta-Cluster-Infrastructure, 10Operations: Mails through deployment-mx SPF & DKIM fails - https://phabricator.wikimedia.org/T87338#4269753 (10Krenair) I've added SPF and DMARC (p=none) records. Haven't done DKIM yet. [04:09:42] !log added SPF and DMARC records to beta.wmflabs.org [04:09:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:28:17] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [05:34:14] lol [05:34:29] simple dns TXT changes [05:34:41] nice one designate [05:53:38] hm this is kind of worrying [05:54:02] it's not normally broken for this long [05:54:36] labs-ns1 resolves it fine but labs-ns0 does not... [05:54:42] there we go [05:58:22] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 0.73 ms [06:01:25] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [06:23:54] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms [06:35:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot-core, 10Wikimedia-Apache-configuration, and 2 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#2189521 (10Legoktm) Here's my hunch...we automatically redirect if ther... [06:51:45] (03PS1) 10Legoktm: Publish pywikibot docs to /$DOC_SUBPATH [integration/config] - 10https://gerrit.wikimedia.org/r/439454 (https://phabricator.wikimedia.org/T132136) [07:01:30] (03CR) 10Legoktm: [C: 032] Publish pywikibot docs to /$DOC_SUBPATH [integration/config] - 10https://gerrit.wikimedia.org/r/439454 (https://phabricator.wikimedia.org/T132136) (owner: 10Legoktm) [07:02:24] (03PS1) 10Legoktm: doc: Point to pywikibot/master/ [integration/docroot] - 10https://gerrit.wikimedia.org/r/439456 (https://phabricator.wikimedia.org/T132136) [07:03:29] (03Merged) 10jenkins-bot: Publish pywikibot docs to /$DOC_SUBPATH [integration/config] - 10https://gerrit.wikimedia.org/r/439454 (https://phabricator.wikimedia.org/T132136) (owner: 10Legoktm) [07:04:27] (03CR) 10Legoktm: [C: 032] doc: Point to pywikibot/master/ [integration/docroot] - 10https://gerrit.wikimedia.org/r/439456 (https://phabricator.wikimedia.org/T132136) (owner: 10Legoktm) [07:04:59] (03Merged) 10jenkins-bot: doc: Point to pywikibot/master/ [integration/docroot] - 10https://gerrit.wikimedia.org/r/439456 (https://phabricator.wikimedia.org/T132136) (owner: 10Legoktm) [07:05:05] (03CR) 10jenkins-bot: doc: Point to pywikibot/master/ [integration/docroot] - 10https://gerrit.wikimedia.org/r/439456 (https://phabricator.wikimedia.org/T132136) (owner: 10Legoktm) [07:05:07] 10Gerrit: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4269796 (10greg) [07:05:53] 10Gerrit: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4269796 (10greg) @Paladox can you help outline what's needed here? [07:08:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot-core, 10Wikimedia-Apache-configuration, and 3 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4269809 (10Legoktm) OK, so there's still a bit of Varnish caching invol... [07:12:08] 10Gerrit: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4269810 (10Legoktm) [07:20:51] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Pywikibot-core, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208#3937263 (10Legoktm) We have a lot of flexibility with jenkins, we could have it run the longer tests in a separate queue (so... [07:56:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot-core, 10Wikimedia-Apache-configuration, and 3 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4269873 (10Dzahn) oh nice, thanks for fixing it this way :) [10:23:32] is it just me or is gerrit not loading? [10:24:50] https://downforeveryoneorjustme.com/gerrit.wikimedia.org not just me [10:25:52] Project beta-code-update-eqiad build #208792: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/208792/ [10:28:57] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:35:00] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:35:22] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:38:30] PROBLEM - Puppet errors on deployment-kafka-main-1 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:38:42] Project beta-code-update-eqiad build #208793: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/208793/ [10:40:59] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [10:42:08] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269936 (10Marostegui) [10:42:33] PROBLEM - Puppet errors on integration-slave-jessie-1003 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:43:45] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:45:27] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269948 (10Marostegui) p:05Triage>03High [10:45:34] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269949 (10greg) Related: The Gerrit upgrade included a migration that created many new git refs. Those are replicated to Phabricator and thus it also had to ingest/index them. [10:46:21] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269950 (10Marostegui) Ah right! I'm from my phone and cannot check what the writes are. Any ETA for that to be finished? [10:47:05] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269951 (10greg) Not sure, @mmodell ? @demon ? [10:48:50] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:48:54] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:50:05] PROBLEM - Puppet errors on deployment-webperf11 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:51:32] Project beta-code-update-eqiad build #208794: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/208794/ [10:52:45] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [10:53:03] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269954 (10Marostegui) Codfw is lagging behind as it cannot cope with the amount of writes. Not a big deal as it is not used, but it is an indicative of how massive it is. It w... [11:04:22] Project beta-code-update-eqiad build #208795: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/208795/ [11:04:36] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269955 (10Paladox) It will be a long while as phab has to parse all the new commits (notedb) we should probaly try to ignore refs/changes/**/meta in phabricator. [11:07:56] PROBLEM - Puppet errors on deployment-kafka-main-2 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:11:00] Project beta-code-update-eqiad build #208796: 04STILL FAILING in 6 min 37 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/208796/ [11:13:46] Yippee, build fixed! [11:13:46] Project beta-code-update-eqiad build #208797: 09FIXED in 45 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/208797/ [11:17:46] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269958 (10Marostegui) >>! In T196840#4269955, @Paladox wrote: > It will be a long while as phab has to parse all the new commits (notedb) we should probaly try to ignore refs/... [11:18:48] RECOVERY - Puppet errors on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [0.0] [11:18:56] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4269961 (10Paladox) Yep days. [11:21:01] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [11:27:46] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:28:27] 10Gerrit: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4269963 (10Paladox) Yep I can, just need to find the config that does the footer. [11:30:00] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [11:30:20] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [11:32:33] RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [11:33:56] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:51] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [11:42:06] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4269969 (10MarcoAurelio) [11:42:57] RECOVERY - Puppet errors on deployment-kafka-main-2 is OK: OK: Less than 1.00% above the threshold [0.0] [11:43:31] RECOVERY - Puppet errors on deployment-kafka-main-1 is OK: OK: Less than 1.00% above the threshold [0.0] [11:45:04] RECOVERY - Puppet errors on deployment-webperf11 is OK: OK: Less than 1.00% above the threshold [0.0] [11:48:54] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:03:54] paladox: around? [12:06:05] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4270016 (10demon) We haven't needed to replicate any refs other than heads and tags since we brought gitiles online... Disable them. Now. And prune them from Phab while we're... [12:06:32] (03PS1) 10MarcoAurelio: Temporary allow Gerrit Managers to own this repository for archiving purposes [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439468 (https://phabricator.wikimedia.org/T186096) [12:14:31] (03Abandoned) 10MarcoAurelio: Temporary allow Gerrit Managers to own this repository for archiving purposes [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439468 (https://phabricator.wikimedia.org/T186096) (owner: 10MarcoAurelio) [12:15:23] (03PS1) 10MarcoAurelio: Mark repository as read only [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439469 [12:16:01] (03PS2) 10MarcoAurelio: Mark repository as read only [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439469 [12:16:32] (03PS3) 10MarcoAurelio: Mark repository as read only [software/tessera] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439469 (https://phabricator.wikimedia.org/T186096) [12:35:52] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot-core, and 2 others: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860#4270050 (10Dvorapa) [12:36:14] 10Release-Engineering-Team (Watching / External), 10Pywikibot-General: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306#4270052 (10Dvorapa) [12:36:17] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot-core, and 2 others: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860#3865572 (10Dvorapa) [12:47:13] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4270059 (10Paladox) But phabricator changed it's behaviour and now clones refs/**. So to fix this we need regex to not clone refs/changes/**/meta. [12:49:08] no_justification oh i wonder if gerrit acl would be useful in rejecting any anon from cloning refs/changes/*/meta [12:50:10] I don't trust gerrit branch-level ACLs. Also: it's perfectly acceptable behavior [12:50:17] branch-level read ACLs, that is [12:51:08] If Phabricator changed to /unconditionally/ fetch all of refs/* and didn't make it configurable, then that was a poor design decision [12:51:23] ah [12:51:25] yeh [12:51:34] no_justification i was trying to look at this months ago [12:51:40] but got lost in how to do it [12:52:03] But yeah, it's not up to gerrit to block refs that clients shouldn't fetch [12:52:08] ok [12:52:20] Rephrase: that clients don't even want [12:52:25] They should be smarter about fetching :) [12:52:27] There we go! [12:52:31] * no_justification needs coffee [12:53:04] heh [12:53:32] no_justification also gerrit's ui froze today (woulden't load) i wonder if it ran out on the amount of files it could open [12:53:43] i have an idea on what could be cause if the answer is yes [12:54:04] Idk. I tried to investigate but it got restarted first [12:54:13] ah ok [12:54:17] Relatedly, came across https://phabricator.wikimedia.org/P7234 [12:54:29] That's the 65 changes that NoteDB couldn't index (because repos were deleted) [12:54:34] yeh [12:54:41] Well, 64 of those 65 pointed to deleted repos..... [12:54:45] So I deleted the changes entries [12:54:48] So those won't error [12:54:53] ok [12:55:01] no_justification i think it would have been jgit gc [12:55:02] #62: 432236 is pointing at a real repo tho [12:55:07] there's a bug that has been fixed [12:55:10] patch for it is https://gerrit-review.googlesource.com/c/gerrit/+/183810 [12:55:15] Nope, those are on repos that /were deleted/ [12:55:21] The changes don't even exist anywhere now [12:55:26] ah ok [12:55:35] Except 432236 [12:55:39] yeh the deleteproject did not remove them from the db heh [12:55:43] Which gives same NoteDB error but seems to exist :) [12:55:48] yeh [12:56:01] So we're 64/65 of the way to being past that bug? :P [12:56:11] heh yeh [12:56:18] no_justification did it say that today? [12:56:21] 98.4% done! [12:56:24] Lol [12:56:26] lol [12:56:42] no_justification i could at least see my account info @ all-users [12:56:45] Those were numbers Daniel pulled from logs. I wasn't near my computer at the time. [12:56:53] Ohhhh, the preferences thing. [12:56:57] I haven't touched that yet [12:57:01] no_justification nope the account info [12:57:08] i could see my full name [12:57:10] and email [12:58:05] any one who uses the group admin ui and finds it buggy can blame me lol [12:58:14] i was the one who did that whole ui [12:58:38] but i made it prettier upstream (audit-log) if you use avatars (also fixed bugs too) [13:07:23] no_justification do you know how to add things to the footer in gwtui? [13:11:28] Um... Like what? [13:12:24] no_justification this https://phabricator.wikimedia.org/T196835 [13:14:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot-core, 10Wikimedia-Apache-configuration, and 3 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4270104 (10Dvorapa) 05Open>03Resolved a:03Legoktm I was not the a... [13:16:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot-core, 10Wikimedia-Apache-configuration, and 3 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4270108 (10Dvorapa) Buut, wait. What if someone goes to `https://doc.wi... [13:19:16] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Pywikibot-core, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208#4270109 (10Dvorapa) I think it would do, yes! [13:26:25] https://github.com/larrycai/gerrit-demo/blob/master/gerrit/etc/GerritSiteFooter.html [13:26:26] oh [13:26:42] now i will need to see how to do this in polygerrit [13:27:52] hmm [13:27:58] https://github.com/larrycai/gerrit-demo/blob/master/gerrit/etc/GerritSiteFooter.html shows above the footer [13:30:55] no_justification https://gerrit.unlegacy-android.org/#/q/status:open lol [13:31:02] they made a nice ui in gwtui heh [13:37:15] There should just be an easy hook to add to those links at the bottom. [14:06:25] https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/439483/ [14:06:27] no_justification ^^ [14:06:32] and that looks like: [14:07:21] https://gerrit.git.wmflabs.org/r/#/dashboard/self [14:07:25] https://gerrit.git.wmflabs.org/r/ [14:08:37] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270155 (10Paladox) ^^ that does it for gwtui only for now, need to figure out how to do it for PolyGerrit. [14:21:14] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270211 (10Paladox) This is what it looks like in gwtui: {F21015415} [14:23:03] no_justification apparently it's not possible to do it in polygerrit unless we can use js to match a specific line in the footer (that's if gerrit-theme.html allows us to do that) [16:41:22] no_justification http://gerrit-wm.wmflabs.org/r/changes/operations%2Fpuppet~439483/detail?O=516714 is returning internal server error [16:41:42] https://gerrit.wikimedia.org/r/changes/operations%2Fpuppet~439483/detail?O=516714 [16:44:24] * paladox wonders why it is a internal error though [16:53:39] PROBLEM - Puppet errors on deployment-puppetmaster03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:02:37] [2018-06-10 17:01:28,564] [HTTP-78] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /r/changes/test~1961/detail?O=516714 [17:02:37] java.lang.IllegalArgumentException: unknown 400000 [17:03:38] RECOVERY - Puppet errors on deployment-puppetmaster03 is OK: OK: Less than 1.00% above the threshold [0.0] [17:03:54] !log also fixed scap install on -zotero01 [17:03:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:04:58] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4269796 (10Legoktm) Is it not possible to have it in the same "Powered by Gerrit..." line? [17:05:21] 10Diffusion: Diffusion commits stuck in 'Importing...' status for too long - https://phabricator.wikimedia.org/T196855#4270440 (10MarcoAurelio) [17:06:27] 10Diffusion: Diffusion commits stuck in 'Importing...' status for too long - https://phabricator.wikimedia.org/T196855#4270450 (10MarcoAurelio) [17:06:48] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270451 (10Paladox) apparently upstream hacked there deployed prod version (i coulden't find a hook to hook into). I am guessing they possibly used js or modified there version.... [17:07:33] 10Diffusion: Diffusion commits stuck in 'Importing...' status for too long - https://phabricator.wikimedia.org/T196855#4270440 (10Paladox) This is due to the gerrit upgrade which created notedb commits in each repo as it is migrating commits and other stuff related to changes (in those projects) into the repo's... [17:07:41] no_justification i filled https://bugs.chromium.org/p/gerrit/issues/detail?id=9226 [17:07:59] 10Diffusion: Inconsistent repo naming - https://phabricator.wikimedia.org/T186910#3959091 (10MarcoAurelio) I'd go with `extension-XXXX`. I've been renaming some to that scheme before knowing about this ticket. [17:10:57] 10Diffusion: Inconsistent repo naming - https://phabricator.wikimedia.org/T186910#4270459 (10MarcoAurelio) This seems related to T135403 as well. [17:16:38] 10Phabricator, 10Patch-For-Review: Phabricator footer links should use Special:MyLanguage for links that support it - https://phabricator.wikimedia.org/T196836#4270466 (10Framawiki) a:03Aklapper [17:17:37] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:19:07] 10Diffusion: To add observe URI we have to disable 6 before doing that - https://phabricator.wikimedia.org/T196856#4270468 (10MarcoAurelio) [17:25:52] no_justification lol that bug only happens on 2.15 not master. [17:26:00] but i doin't see any fix on master [17:26:03] for this [17:27:37] (though no one should be getting that error) [17:27:58] i am using server.go to see how the master ui will look like on gerrit on gerrit.wikimedia.org [18:33:06] about time i managed to figure out how to use gerrit-theme.html now [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:40:31] woo hoo and figured out how to do it in polygerrit now [18:40:33] the footer! [18:40:40] using javascript magic [18:42:52] 10Diffusion: Diffusion commits stuck in 'Importing...' status for too long - https://phabricator.wikimedia.org/T196855#4270512 (10Aklapper) Related to / potential duplicate of {T196840} ? [18:45:11] 10Diffusion: Diffusion commits stuck in 'Importing...' status for too long - https://phabricator.wikimedia.org/T196855#4270515 (10Paladox) Yep that is related to that task. [19:21:31] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Pywikibot-core, 10Pywikibot-tests: Magul's quick tests doesn't run anymore - https://phabricator.wikimedia.org/T186208#4270536 (10Xqt) >>! In T186208#4269825, @Legoktm wrote: > Would that meet the current needs? +1 [19:47:08] https://phabricator.wikimedia.org/rESHMeca6da77316413a1e2c8347d76817ba184af6f8e <-- this doesn't make any sense [19:49:19] hauskatze that's a notedb comment [19:49:33] that's meta data that use to be in the db [19:49:37] but is now stored in the repo [19:49:40] git commit style [19:50:10] well, I still don't see much sense to count those :) [19:50:31] no wonder why Phab repos now have 100k commits [19:50:50] yep, but phabricator is the one that does not allow us to configure that though :) [19:51:18] heh [20:04:53] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270584 (10Paladox) this is what the polygerrit change looks like {F21243950} [20:33:31] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270594 (10Paladox) @legoktm actually i can do what i did for polygerrit using javascript! [20:44:47] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270605 (10greg) >>! In T196835#4270594, @Paladox wrote: > @legoktm actually i can do what i did for polygerrit using javascript! inb4: "if i don't have JS enabled am I bound by... [20:46:45] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270606 (10Paladox) hmm, gerrit should not even load if you have javascript disabled :). Expecially for polygerrit which is built manly around javascript and html. [20:52:04] 10Gerrit, 10Patch-For-Review: Add a link to the Code of Conduct to the footer of Gerrit - https://phabricator.wikimedia.org/T196835#4270613 (10Paladox) @greg just tested and gerrit gwtui throws "Gerrit requires a JavaScript enabled browser.". and polygerrit has a whitescreen. [21:12:58] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4269209 (10Paladox) [21:14:49] 10Gerrit, 10Release-Engineering-Team: Gerrit plugin "zuul" failed to load - https://phabricator.wikimedia.org/T195176#4270641 (10Paladox) Now that we have gerrit 2.15, polygerrit ui is alot more usable now, so i recommend you use that ui :) [21:18:22] 10Gerrit: Create a code coverage plugin for polygerrit - https://phabricator.wikimedia.org/T183760#4270653 (10Paladox) This can be done now that we have upgraded to 2.15 :). [21:40:50] 10Gerrit, 10Developer-Relations, 10GitHub-Mirrors, 10Repository-Admins, and 3 others: Add CODE_OF_CONDUCT.md to Wikimedia repositories - https://phabricator.wikimedia.org/T165540#4270658 (10Tgr) >>! In T165540#4268283, @Nemo_bis wrote: > We generally use plain text files (the classic `COPYING`, `README` et... [21:45:47] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Ignore refs/changes/**/**/meta - https://phabricator.wikimedia.org/T196860#4270661 (10Paladox) [21:45:54] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Ignore refs/changes/**/**/meta - https://phabricator.wikimedia.org/T196860#4270671 (10Paladox) p:05Triage>03High [21:48:40] https://github.com/wikimedia/phabricator/blob/915c010e61e6c9faa7142541c1837fe1cb4eb634/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php#L417 [21:58:21] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Ignore refs/changes/**/**/meta - https://phabricator.wikimedia.org/T196860#4270672 (10Paladox) [21:58:24] 10Gerrit, 10Phabricator, 10DBA, 10Operations: Massive increase of writes in m3 section - https://phabricator.wikimedia.org/T196840#4270673 (10Paladox) [22:09:15] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Ignore refs/changes/**/**/meta - https://phabricator.wikimedia.org/T196860#4270678 (10Paladox) [22:11:40] paladox: yeah, it ("inb4 javascript disabled") was just me being snarky on the weekend :) [22:11:51] heh [22:12:38] greg-g i would have done it without js but upstream hacked there version (which ment that it wasen't really supported) :) [22:27:00] oh i think i found the proper fix for the large importing of refs [22:27:08] testing currently [22:29:48] yep the fix works [22:29:54] ammending my prevous patch [22:33:46] https://phabricator.wikimedia.org/D1067 [22:34:43] 10Diffusion: Diffusion commits stuck in 'Importing...' status for too long - https://phabricator.wikimedia.org/T196855#4270698 (10Paladox) [22:43:11] legoktm hi, around? [22:43:21] paladox: kinda, what's up? [22:43:22] does saving in your preference work? :) [22:43:33] (the issue you reported a few days ago) [22:44:09] oops, I never tried again [22:44:11] lemme do that now [22:44:26] ok :) [22:44:28] thanks [22:44:36] paladox: nope :( [22:44:39] hmm [22:44:42] ok [22:44:47] * paladox creates a task [22:44:58] specifically, I'm using polygerrit, on https://gerrit.wikimedia.org/r/settings/ unchecking "Size" under Change Table Columns [22:47:10] legoktm i am thinking a corrupt file [22:47:13] under your ref [22:47:16] in All-users [22:47:41] 10Gerrit, 10Release-Engineering-Team: Some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196862#4270699 (10Paladox) [22:47:49] 10Gerrit, 10Release-Engineering-Team: Some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196862#4270709 (10Paladox) p:05Triage>03Unbreak! [22:47:51] legoktm ^^ [22:48:17] i have no access to logs so cannot investigate until someone posts the logs :). [22:51:40] paladox: the logs should be in logstash right? [22:51:45] legoktm yeh [22:51:51] i think so [22:54:37] though i noticed it dosen't log everything (if it's a large stack trace though) [22:55:05] paladox: uh, do you know what type/host I should be looking for? [22:55:17] legoktm the type is log4j [22:55:38] there we go [22:55:48] (might be nice to document that on https://wikitech.wikimedia.org/wiki/Gerrit) [22:57:06] legoktm done https://wikitech.wikimedia.org/wiki/Gerrit#Logstash [22:58:04] I don't see anything relevant [22:58:11] the only 3 entries from the last 15 minutes are all about StackOverflowError prettifying /r/plugins/gitiles/operations/mediawiki-config/+blame/master/wmf-config/interwiki-labs.php?blame=1 [22:58:11] priority [22:58:25] hmm [22:58:33] and, ty for documenting <3 [22:58:38] your welcome :) [22:58:50] i guess gerrit did not log the error you got [22:58:56] i wish gerrit supported gelf [23:00:13] but it would have log it locally [23:00:30] on cobalt you mean? [23:00:33] yep [23:00:40] gelf would fix this problem [23:00:44] 10Gerrit, 10Release-Engineering-Team: Some users are reporting problems saving there preferences - https://phabricator.wikimedia.org/T196862#4270699 (10Legoktm) [15:44:58] specifically, I'm using polygerrit, on https://gerrit.wikimedia.org/r/settings/ unchecking "Size" under Change Table Columns I... [23:01:20] legoktm do you have type to clone https://gerrit.wikimedia.org/r/admin/projects/All-Users ? [23:01:31] and then edit .git/config (refs/heads -> refs/) and git pull [23:01:40] that's where the problem would likley be [23:01:46] it has your preferences file [23:01:52] and account.config [23:02:05] Cloning into 'All-Users'... [23:02:05] fatal: remote error: admin/projects/All-Users unavailable [23:02:35] legoktm git clone ssh://legoktm@gerrit.wikimedia.org:29418/All-Users [23:02:51] also this https://gerrit.wikimedia.org/r/admin/projects/All-Users is a polygerrit url :) [23:02:55] ah [23:03:30] you should be able to git checkout origin/users/self once you fixed .git/config to clone all refs [23:03:36] only clones refs you can see [23:03:51] is there no web viewer for this? [23:04:07] nope [23:04:12] for security reasons i think [23:04:39] so I did [23:04:40] [remote "origin"] [23:04:41] url = https://gerrit.wikimedia.org/r/All-Users [23:04:41] fetch = +refs/*:refs/remotes/origin/* [23:04:43] in my .git/config [23:04:45] yep [23:04:57] did the git pull [23:04:58] km@km-pt ~/g/a/All-Users> git checkout origin/users/self [23:04:58] error: pathspec 'origin/users/self' did not match any file(s) known to git. [23:05:04] oh i see now [23:05:14] you need to use https (with username@) or ssh [23:05:17] also the pull only had: [23:05:18] From https://gerrit.wikimedia.org/r/All-Users [23:05:18] * [new ref] refs/changes/39/439439/1 -> origin/changes/39/439439/1 [23:05:18] * [new ref] refs/changes/39/439439/meta -> origin/changes/39/439439/meta [23:05:18] * [new ref] refs/meta/config -> origin/meta/config [23:05:24] ah [23:05:29] change the url to something like [23:05:33] url = ssh://legoktm@gerrit.wikimedia.org:29418/All-Users [23:05:44] needs ssh or the https url with username@ [23:05:51] so it can authenticate against gerrit [23:05:57] and let you see your ref [23:07:27] ok nice [23:07:37] should I pastebin my preferences.config for you to see? [23:07:53] yes please :) [23:08:16] https://paste.fedoraproject.org/paste/9XLIvINBBH8dd21ddSA6lA [23:08:40] thanks! [23:09:05] * legoktm brb ~5 min [23:09:10] ok [23:11:55] nothing looks wrong with your file either hmm [23:12:27] what about account.config ? [23:12:32] mine looks like: [23:12:51] https://phabricator.wikimedia.org/P7237 [23:13:02] paladox: https://paste.fedoraproject.org/paste/Tinwf-4~xlW2JUfZqTsjYg just no status [23:13:09] ok thanks [23:13:22] they all look ok [23:13:23] so hmm [23:19:39] paladox: anyways, not a huge deal for me atm. probably whenever someone with shell access can look at the log entry it'll be more obvious? [23:19:47] yeh [23:19:55] legoktm though it's happening to other users too [23:20:04] oh :( [23:20:06] another user pm'ed me a few days ago