[00:04:23] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [00:06:33] RECOVERY - Puppet errors on integration-slave-jessie-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [00:07:48] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [00:10:14] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<33.33%) [00:11:57] RECOVERY - Puppet errors on deployment-kafka-main-2 is OK: OK: Less than 1.00% above the threshold [0.0] [00:12:29] RECOVERY - Puppet errors on deployment-kafka-main-1 is OK: OK: Less than 1.00% above the threshold [0.0] [00:14:57] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [00:17:47] RECOVERY - Puppet errors on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [0.0] [00:17:55] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:19:05] RECOVERY - Puppet errors on deployment-webperf11 is OK: OK: Less than 1.00% above the threshold [0.0] [00:21:26] (03CR) 10MaxSem: [C: 032] Add "Generic.PHP.LowerCaseType" to ruleset [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439298 (owner: 10Legoktm) [00:22:56] (03CR) 10MaxSem: [C: 032] Use "PSR12.Keywords.ShortFormTypeKeywords" in place of custom sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439299 (owner: 10Legoktm) [00:26:47] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [00:29:00] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [00:29:20] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:32:57] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:39:21] (03PS1) 10SamanthaNguyen: Archive TopLists extension [integration/config] - 10https://gerrit.wikimedia.org/r/439370 (https://phabricator.wikimedia.org/T196786) [01:14:29] (03PS1) 10SamanthaNguyen: Archive EditPageTracking extension [integration/config] - 10https://gerrit.wikimedia.org/r/439377 (https://phabricator.wikimedia.org/T190894) [01:55:13] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: Name or service not known [01:57:28] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [02:02:54] uh wtf [02:03:18] LGTM [02:03:53] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 2.70 ms [02:12:23] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#4268622 (10Krenair) It looks like @joe has made and merged patches that essentially obsolete tho... [02:13:30] !log shut down old deployment-redis01 and deployment-redis02 instances T179371 [02:13:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:13:34] T179371: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371 [02:15:19] (03CR) 10MaxSem: [C: 032] Upgrade squizlabs/php_codesniffer to 3.3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439297 (owner: 10Legoktm) [02:15:47] PROBLEM - Host deployment-redis02 is DOWN: CRITICAL - Host Unreachable (10.68.16.231) [02:15:51] PROBLEM - Host deployment-redis01 is DOWN: CRITICAL - Host Unreachable (10.68.16.177) [02:16:30] (03Merged) 10jenkins-bot: Upgrade squizlabs/php_codesniffer to 3.3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439297 (owner: 10Legoktm) [02:16:36] (03Merged) 10jenkins-bot: Add "Generic.PHP.LowerCaseType" to ruleset [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439298 (owner: 10Legoktm) [02:16:38] (03Merged) 10jenkins-bot: Use "PSR12.Keywords.ShortFormTypeKeywords" in place of custom sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439299 (owner: 10Legoktm) [02:17:00] (03CR) 10jenkins-bot: Upgrade squizlabs/php_codesniffer to 3.3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439297 (owner: 10Legoktm) [02:17:18] !log shut down old deployment-dumps-puppetmaster instance (replaced with a newer stretch instance), emailed ariel [02:17:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:17:21] (03CR) 10jenkins-bot: Add "Generic.PHP.LowerCaseType" to ruleset [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439298 (owner: 10Legoktm) [02:17:31] so I've got 4 instances on my list to get rid of now [02:17:41] (03CR) 10jenkins-bot: Use "PSR12.Keywords.ShortFormTypeKeywords" in place of custom sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/439299 (owner: 10Legoktm) [02:18:02] PROBLEM - Host deployment-dumps-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.21.153) [02:18:42] deployment-puppetmaster02, deployment-dumps-puppetmaster, deployment-redis01, deployment-redis02 [02:19:11] wtf how is puppetmaster02 running [02:20:03] !log stopping deployment-puppetmaster02 again, looks like it was automatically booted by novaadmin after security patches a couple days ago [02:20:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:24:21] PROBLEM - Host deployment-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (10.68.21.200) [02:24:46] Reedy, ssh deployment-cache-text04 sudo openssl x509 -in ~krenair/acme-v2-test/out.pem -noout -text [02:25:19] 10Gerrit: Please rewrite sync-with-gerrit.py to use Gerrit REST API - https://phabricator.wikimedia.org/T194318#4195590 (10demon) Every user has SSH access to the server. And you'd need to be a user to do the commit to the repo....who's being excluded here? But I don't disagree, the REST api is always an improv... [02:30:16] 10Gerrit, 10Patch-For-Review: Switch to mariadb java connector - https://phabricator.wikimedia.org/T176164#4268643 (10demon) I think we can just decline this. NoteDB works around the problem and the underlying bug leading to this has gone away too. [02:31:38] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927#4268645 (10Krenair) ```krenair@deployment-cache-text04:~/acme-v2-test$ openssl x509 -in out.pem -noout -text Certificate: Data: Version: 3 (0x2... [03:06:15] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: Name or service not known [03:08:51] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [03:10:17] this is very interesting [03:10:56] DNS fail [03:11:01] yeah [03:11:10] timing is similar to when my TXT record creations/deletions are going on [03:12:54] but there's nothing changing about the *.beta.wmflabs.org A record so there should be no outage [03:13:52] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 3.62 ms [03:15:40] caught it red-handed [03:16:32] MaxSem, https://phabricator.wikimedia.org/P7232 [03:17:18] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [03:23:51] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 2.24 ms [03:24:18] -> https://phabricator.wikimedia.org/T196797 [04:43:30] paladox: on gerrit, when I view a list of changes, I can only see checks and crosses for CR/V, not the users who left those votes [04:44:36] paladox, no_justification: I tried editing my preferences on Gerrit and I got "Internal server error" [05:30:14] (03CR) 10Hashar: [C: 032] Archive TopLists extension [integration/config] - 10https://gerrit.wikimedia.org/r/439370 (https://phabricator.wikimedia.org/T196786) (owner: 10SamanthaNguyen) [05:32:19] (03Merged) 10jenkins-bot: Archive TopLists extension [integration/config] - 10https://gerrit.wikimedia.org/r/439370 (https://phabricator.wikimedia.org/T196786) (owner: 10SamanthaNguyen) [05:38:45] 10Gerrit, 10Release-Engineering-Team: Change rebase causes a Missing blob e959c00909e3b4ae11c26a58616a2d699883dcf3 - https://phabricator.wikimedia.org/T196800#4268782 (10hashar) [05:51:35] 10Continuous-Integration-Config, 10Patch-For-Review, 10Test-Coverage: PHP test coverage Jenkins report should congratulate success rather than focusing on failure - https://phabricator.wikimedia.org/T192853#4268798 (10hashar) 05Open>03Resolved a:03Legoktm [08:40:17] 10Gerrit: Please rewrite sync-with-gerrit.py to use Gerrit REST API - https://phabricator.wikimedia.org/T194318#4268936 (10MarcoAurelio) ``` $ ssh -a -p 29418 maurelio@gerrit.wikimedia.org **** Welcome to Gerrit Code Review **** Hi MarcoAurelio, you have successfully connected over SSH. Unfortunat... [08:45:46] 10Gerrit, 10Release-Engineering-Team: Change rebase causes a Missing blob e959c00909e3b4ae11c26a58616a2d699883dcf3 - https://phabricator.wikimedia.org/T196800#4268782 (10MarcoAurelio) This is probably because of the submodule commit being updated. Manual rebase has helped me in other instances (and that's why... [09:07:23] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:07:26] (03PS1) 10MarcoAurelio: Archive the VectorV2 skin [skins/VectorV2] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439433 (https://phabricator.wikimedia.org/T196169) [09:08:23] (03PS2) 10MarcoAurelio: Archive the VectorV2 skin [skins/VectorV2] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439433 (https://phabricator.wikimedia.org/T196169) [10:48:21] 10Gerrit: Please rewrite sync-with-gerrit.py to use Gerrit REST API - https://phabricator.wikimedia.org/T194318#4269019 (10demon) You can't have an interactive shell. But sync-with-gerrit doesn't do that....try appending `gerrit ls-projects` to that [11:04:55] legoktm: hmm, I wonder why it 500 for you [11:24:21] legoktm: could you file a task [11:24:32] Other users I think reported that too [11:24:44] I am guessing that is notedb. [11:25:03] Accounts are stored on notedb [11:25:08] All-Users [11:31:34] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10GitHub-Mirrors, 10Pywikibot-core, and 2 others: AppVeyor test not running since months - https://phabricator.wikimedia.org/T183860#4269024 (10Dvorapa) p:05Lowest>03Normal [12:27:27] 10Gerrit, 10GitHub-Mirrors, 10Pywikibot-core, 10Repository-Admins: Grant access to GitHub's wikimedia/pywikibot repository to current Pywikibot developer base - https://phabricator.wikimedia.org/T196810#4269100 (10Dvorapa) [12:51:06] (03PS1) 10Paladox: test [All-Users] (refs/users/65/1665) - 10https://gerrit.wikimedia.org/r/439439 [12:51:18] oh wow [12:51:22] that worked! [12:51:36] legoktm try editing your branch in All-Users ^^ [12:53:00] so you clone All-Users [12:53:03] edit .git/config [12:53:11] and replace refs/heads with refs/* [12:53:47] it will then show you refs/users/self and refs/usernumber/user_number [12:58:04] (03Abandoned) 10Paladox: test [All-Users] (refs/users/65/1665) - 10https://gerrit.wikimedia.org/r/439439 (owner: 10Paladox) [12:58:11] Phab activity feed is almost useless now [12:58:14] Showing every gerrit PS [12:59:10] 10Phabricator (Upstream), 10Developer-Wishlist (2017), 10Upstream: Cannot disable "Notify" for token award in phabricator - https://phabricator.wikimedia.org/T91289#4269119 (10Aklapper) Upstream https://secure.phabricator.com/T7468 got merged into https://secure.phabricator.com/T10448 which is resolved. [13:01:41] 10Phabricator, 10Community-Liaisons, 10Developer-Relations, 10Developer-Wishlist (2017), 10Goal: Consolidate the many tech events calendars in Phabricator's calendar - https://phabricator.wikimedia.org/T1035#4269121 (10Aklapper) I'd propose to revert 2015's T1035#1579199 and go back to the original task... [13:02:19] 10Gerrit: Cannot access self dashboard after 2.15 upgrade - https://phabricator.wikimedia.org/T196768#4269122 (10Paladox) 05Open>03Resolved a:03Paladox We fixed it by doing a offline reindex which was quicker. [13:02:26] 10Gerrit: Cannot access self dashboard after 2.15 upgrade - https://phabricator.wikimedia.org/T196768#4269125 (10Paladox) a:05Paladox>03None [13:03:01] 10Gerrit, 10Upstream: Polygerrit search dropdown does not list all projects - https://phabricator.wikimedia.org/T188842#4269126 (10Paladox) 05Open>03Resolved [13:03:15] 10Gerrit, 10Upstream: Polygerrit search dropdown does not list all projects - https://phabricator.wikimedia.org/T188842#4021255 (10Paladox) We have upgraded to 2.15 now. [13:04:26] 10Gerrit: Gerrit: autocomplete to add reviewers slow - https://phabricator.wikimedia.org/T183234#4269129 (10Paladox) Please re try as we have upgraded to 2.15 now, there should be performance improvements in PolyGerrit ui, upstream have begun removing gwtui by removing parts of gwtui from the build process so on... [13:05:25] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#4269130 (10Paladox) 05Open>03Resolved a:03Paladox We are now using NoteDB as off when ever it migrated the last change. All new changes should be using notedb from now. [13:05:31] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#4269133 (10Paladox) a:05Paladox>03None [13:07:49] 10Gerrit: Enable Gerrit feature to add comment when people add reviewers to a patch - https://phabricator.wikimedia.org/T168030#3353627 (10Paladox) [13:07:51] 10Gerrit, 10Release-Engineering-Team (Someday): Update gerrit to 2.15.2 - https://phabricator.wikimedia.org/T177201#4269134 (10Paladox) 05Open>03Resolved a:03demon [13:08:48] 10Gerrit, 10Release-Engineering-Team (Someday), 10Operations, 10Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#4269142 (10Paladox) 05stalled>03Resolved We are now on 2.15 and just tested and emoji's work now! [13:09:34] 10Gerrit, 10Developer-Wishlist (2017), 10Patch-For-Review, 10Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#4269148 (10Paladox) 05Open>03Resolved a:05Paladox>03None hashtags have been enabled now since notedb migrator kicked in. [13:09:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Pywikibot-core, 10Wikimedia-Apache-configuration, and 2 others: Pywikibot documentation showing broken directory listing - https://phabricator.wikimedia.org/T132136#4269151 (10Dvorapa) >>! In T132136#4215978, @Dzahn wrote: > I thought t... [13:11:02] 10Deployments, 10Gerrit, 10ReleaseTaggerBot, 10WorkType-NewFunctionality: Deployment status indicator for gerrit patches - https://phabricator.wikimedia.org/T88136#4269153 (10Paladox) With PolyGerrit we could develop a plugin that display deployment status right above the file list. Would need some kind of... [13:14:27] 10Gerrit, 10Documentation, 10Need-volunteer: Update Wikimedia's Gerrit documentation for new user interface in Gerrit 2.15 - https://phabricator.wikimedia.org/T179759#4269158 (10Paladox) [13:16:57] 10Gerrit, 10Zuul: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148#4269162 (10Paladox) We have upgraded to 2.15 now, and clicking on the "Show comments only" button, it hides the Uploaded patchset . [13:18:47] PROBLEM - Puppet errors on deployment-maps03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:21:38] 10Gerrit, 10Patch-For-Review: Switch to mariadb java connector - https://phabricator.wikimedia.org/T176164#4269163 (10Paladox) 05Open>03declined Yep, in 2.16 / 3.0 groups will be migrated to notedb so only one last piece would be using the db (which i forget which one) [14:01:12] https://gerrit.wikimedia.org/r/admin/groups will become alot faster in 2.16 / 3.0 i think :) [14:22:24] 10Continuous-Integration-Config, 10Pywikibot-core, 10Documentation, 10Pywikibot-Documentation: Jenkins should index all 'FixMe' or 'Todo' in pywikibot codebase - https://phabricator.wikimedia.org/T67172#699118 (10Dvorapa) This would be cool, but Sphinx can only index those in docs. But what about those in... [14:46:13] 10Gerrit: Gerrit: autocomplete to add reviewers slow - https://phabricator.wikimedia.org/T183234#4269243 (10Volans) From a quick test the slowest one letter search was ~1s and was for less common letters like `z` or `q`. As of now I cannot repro the issue, feel free to resolve the task if you think that the new... [14:48:15] 10Gerrit: Gerrit: autocomplete to add reviewers slow - https://phabricator.wikimedia.org/T183234#4269247 (10Paladox) 05Open>03Resolved Ok :). PolyGerrit is getting some more performance improvements upstream anyways :) [14:54:23] (03PS1) 10Kosarajugopikrishna: Insert the description of the change. [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439445 [14:54:36] 10Gerrit, 10Developer-Relations, 10Documentation: Update Wikimedia's Gerrit documentation for new user interface in Gerrit 2.15 - https://phabricator.wikimedia.org/T179759#4269270 (10Aklapper) [14:54:38] 10Gerrit, 10Developer-Relations, 10Documentation: Update Wikimedia's Gerrit documentation for new user interface in Gerrit 2.15 - https://phabricator.wikimedia.org/T179759#3735152 (10Aklapper) [14:55:58] 10Gerrit, 10Developer-Relations, 10Documentation: Update Wikimedia's Gerrit documentation for new user interface in Gerrit 2.15 - https://phabricator.wikimedia.org/T179759#4269274 (10Paladox) [15:17:05] Reedy yeh [15:17:17] that's due to notedb [15:17:27] so all comments over the years are being made a git comment [15:18:01] which would mean phabricator is parsing alot of notedb commits [15:44:00] 10Gerrit, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4269308 (10Framawiki) [15:46:13] 10Gerrit, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4269309 (10Framawiki) Proposition of summary for #user-notice / tech news: New users to Wikimedia' Gerrit instance will have the new interface enabled by default as opt-out mode. Every user... [15:59:41] * paladox backports fixes to the group ui if possible :) [15:59:56] well now i am getting conflicts due to the rename from project -> repo [16:09:37] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:24:45] (03PS5) 10Reedy: Remove `composer dump-autoload --optimize` [integration/jenkins] - 10https://gerrit.wikimedia.org/r/394907 (https://phabricator.wikimedia.org/T181940) [16:33:45] done here https://gerrit-review.googlesource.com/c/gerrit/+/183850 :) [18:10:04] 10Gerrit, 10Zuul: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148#4269388 (10Tgr) 2.15 looks awesome! Now we only need to tag `jenkins-bot`, `BarryTheBrowserTestBot` and `Cindy-the-browser-test-bot` as bots. (There are some other bot gerrit accounts: Jenkins-mwext-s... [18:14:56] 10Gerrit, 10Zuul: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148#4269392 (10Tgr) gerrit-review.googlesource.com has a "Comment threads" mode, which adds another level of awesome. Is that a version difference? They are also on 2.15.2. [18:31:44] 10Gerrit, 10Zuul: Allow hiding of non-discussion comments in Gerrit - https://phabricator.wikimedia.org/T48148#4269395 (10Paladox) @tgr upstream use the master branch is has alot of new features to polygerrit. this will be in 2.16 / 3.0. Upstream may be planning on branching a new release in the next couple o... [18:33:42] 10Gerrit, 10Patch-For-Review, 10User-notice: Make PolyGerrit the default ui - https://phabricator.wikimedia.org/T196812#4269396 (10Paladox) [20:09:49] PROBLEM - SSH on deployment-deploy-01 is CRITICAL: Connection refused [20:15:40] 10Gerrit, 10Release-Engineering-Team: Change rebase causes a Missing blob e959c00909e3b4ae11c26a58616a2d699883dcf3 - https://phabricator.wikimedia.org/T196800#4269525 (10hashar) 05Open>03Resolved a:03hashar Solved by rebasing the change :] [20:18:13] (03CR) 10Hashar: [V: 032 C: 032] Archive the VectorV2 skin [skins/VectorV2] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/439433 (https://phabricator.wikimedia.org/T196169) (owner: 10MarcoAurelio) [20:43:50] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:00:58] !log Temporarily substituting certificates on deployment-cache-text04 for certs generated from T182927 to test [21:01:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:01:02] T182927: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 [21:02:15] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927#4269555 (10Krenair) ```alex@alex-laptop:~$ openssl s_client -connect deployment.wikimedia.beta.wmflabs.org:443 | openssl x509 -text -noout depth=2 O = Digi... [21:12:24] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:18:19] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927#4269578 (10Krenair) I've been comparing `openssl s_client -connect meta.wikimedia.org:443 2>&1 | openssl x509 -text -noout | grep DNS: | sed -e 's/^ *//' |... [21:20:46] I think that puppet error on -cache-text04 is actually unrelated to my cert work today [21:21:08] I think that might be related to how I set up /var/lib/puppet/volatile on the new stretch puppetmaster [21:21:47] specifically the error is Error: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Permission denied @ rb_sysopen - /var/lib/puppet/volatile/GeoIP/.geoipupdate.lock [21:30:23] 10Beta-Cluster-Infrastructure: Secure deployment-prep sudo access to prevent privilege escalation by dns-manager credentials - https://phabricator.wikimedia.org/T190781#4269579 (10Krenair) [21:32:24] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [21:35:18] Krenair happens to me [21:35:23] easy fix is to remove the fix [21:35:31] uh i mean [21:35:33] remove the file [21:35:36] on the puppet master [21:35:43] it should regenerate it's self i think [21:35:44] yeah pretty sure I did that once :/ [21:36:13] removed anyway [21:43:26] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:44:43] 10Release-Engineering-Team (Watching / External), 10Pywikibot-General: Share Appveyor account credentials with Release Engineering - https://phabricator.wikimedia.org/T104306#1413120 (10Dvorapa) Finally Pywikibot team went to a phase when the AppVeyor builds are not working and AppVeyor account credentials are... [21:51:43] * paladox wonders how easy it would be to display https://tools.wmflabs.org/versions/ on gerrit changes if the branch and project matches wmf [21:54:39] 10Gerrit: Gerrit upload-pack send ALL references causing massive network I/O on common operations - https://phabricator.wikimedia.org/T103990#4269587 (10Paladox) google has made git protocol v2 and open sourced it and will be available in 2.18. gerrit jgit upstream is gaining support for this new protocol. http... [22:08:55] 10Beta-Cluster-Infrastructure, 10Incident-20160126-WikimediaDomainRedirection, 10Staging, 10Patch-For-Review, 10Wikimedia-Incident: Rework beta apache config - https://phabricator.wikimedia.org/T1256#4269588 (10Krenair) It looks like everybody dropped the ball for actually getting these reviewed. The nex... [22:35:38] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Technical-Debt: Set up LVS in beta like prod - https://phabricator.wikimedia.org/T196662#4269608 (10Krenair) (patch is just an old thing from when I last tried LVS inside labs) [22:49:55] (03Abandoned) 10Paladox: Update composer to 1.4.3 [integration/composer] - 10https://gerrit.wikimedia.org/r/395898 (https://phabricator.wikimedia.org/T125343) (owner: 10Paladox) [22:54:36] legoktm does saving your preference work now? :) [23:04:46] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:07:11] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:08:37] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:11:02] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:12:00] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:14:08] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:15:15] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:17:21] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:17:21] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:18:18] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:21:14] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:22:10] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:25:21] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:27:36] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:28:43] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:29:33] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:30:13] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:33:58] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:36:08] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:37:06] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:55:39] wat [23:56:55] oh scap downgrades [23:57:01] did someone do something [23:58:41] krenair@deployment-tin:~$ apt-cache policy scap [23:58:41] scap: [23:58:41] Installed: 3.9.0-1+0~20180515171834.349~1.gbpcef62d [23:58:41] Candidate: 3.8.2-1+0~20180607230422.353~1.gbp2bb4cc [23:58:41] Version table: [23:58:43] *** 3.9.0-1+0~20180515171834.349~1.gbpcef62d 0 [23:58:45] 100 /var/lib/dpkg/status [23:58:47] 3.8.2-1+0~20180607230422.353~1.gbp2bb4cc 0 [23:58:49] 1500 http://deployment-tin.deployment-prep.eqiad.wmflabs/repo/ jessie-deployment-prep/main amd64 Packages [23:59:20] those puppet error hosts are all complaining about that scap downgrade ^ [23:59:35] dunno how that apt repo works exactl [23:59:36] exactly