[00:00:39] If you see "invalid distance too far back" this means something in that repo (package.json, or castor cache, or package-lock) is triggering the known zlib bug in Node 10 / Debian 9. Solution is to revert the zuul config back for that repo to use Node 6 or whatever the job config was before the recent change. [00:06:37] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10Jdforrester-WMF) [00:07:00] (03PS1) 10Legoktm: Use mediawiki-phan:0.1.12 for mediawiki-core-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/498271 [00:07:13] (03CR) 10Legoktm: [C: 03+2] Use mediawiki-phan:0.1.12 for mediawiki-core-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/498271 (owner: 10Legoktm) [00:10:47] (03Merged) 10jenkins-bot: Use mediawiki-phan:0.1.12 for mediawiki-core-php70-phan-docker [integration/config] - 10https://gerrit.wikimedia.org/r/498271 (owner: 10Legoktm) [00:22:53] (03PS1) 10Jforrester: Follow-up 8c5d25631: Revert Timo's inclusion of changes to VE [integration/config] - 10https://gerrit.wikimedia.org/r/498276 (https://phabricator.wikimedia.org/T215562) [00:29:51] (03CR) 10Krinkle: [C: 03+2] Follow-up 8c5d25631: Revert Timo's inclusion of changes to VE [integration/config] - 10https://gerrit.wikimedia.org/r/498276 (https://phabricator.wikimedia.org/T215562) (owner: 10Jforrester) [00:30:53] (03CR) 10Jforrester: [C: 04-1] "Still used, after a revert in I9ad1b3b0f:" [integration/config] - 10https://gerrit.wikimedia.org/r/496894 (owner: 10Jforrester) [00:31:27] (03Merged) 10jenkins-bot: Follow-up 8c5d25631: Revert Timo's inclusion of changes to VE [integration/config] - 10https://gerrit.wikimedia.org/r/498276 (https://phabricator.wikimedia.org/T215562) (owner: 10Jforrester) [00:36:19] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/498276 / T215562) [00:36:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:36:22] T215562: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 [00:36:44] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/498276 / T218963 [00:36:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:36:46] T218963: CI outage for mediawiki/extensions/VisualEditor – job "mwext-node10-rundoc-docker" always fails and we can't merge code due to repo configuration - https://phabricator.wikimedia.org/T218963 [00:39:55] James_F: good now? [00:40:18] Krinkle: Testing. [00:42:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10VisualEditor: CI outage for mediawiki/extensions/VisualEditor – job "mwext-node10-rundoc-docker" always fails and we can't merge code due to repo configuration - https://phabricator.wikimedia.org/T218963 (10Jdforrester-WMF) a:0... [00:44:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10VisualEditor, 10User-Ryasmeen: CI outage for mediawiki/extensions/VisualEditor – job "mwext-node10-rundoc-docker" always fails and we can't merge code due to repo configuration - https://phabricator.wikimedia.org/T218963 (10Jdf... [00:47:14] Krinkle: Yes. [01:46:20] 10Diffusion, 10Striker, 10Patch-For-Review: Diffusion repository creation fails via toolsadmin - https://phabricator.wikimedia.org/T182142 (10bd808) 05Open→03Resolved [02:48:54] (03PS1) 10Legoktm: Actually run phan docker image with /cache mounted [integration/config] - 10https://gerrit.wikimedia.org/r/498295 (https://phabricator.wikimedia.org/T217479) [02:49:44] (03CR) 10jerkins-bot: [V: 04-1] Actually run phan docker image with /cache mounted [integration/config] - 10https://gerrit.wikimedia.org/r/498295 (https://phabricator.wikimedia.org/T217479) (owner: 10Legoktm) [02:55:06] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [02:56:49] (03PS1) 10Legoktm: Install php-imagick in mediawiki-phan container [integration/config] - 10https://gerrit.wikimedia.org/r/498297 [02:57:45] (03CR) 10jerkins-bot: [V: 04-1] Install php-imagick in mediawiki-phan container [integration/config] - 10https://gerrit.wikimedia.org/r/498297 (owner: 10Legoktm) [03:02:12] (03PS2) 10Legoktm: Actually run phan docker image with /cache mounted [integration/config] - 10https://gerrit.wikimedia.org/r/498295 (https://phabricator.wikimedia.org/T217479) [03:02:14] (03PS2) 10Legoktm: Install php-imagick in mediawiki-phan container [integration/config] - 10https://gerrit.wikimedia.org/r/498297 [03:03:53] PROBLEM - Puppet staleness on deployment-mediawiki-09 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:05:39] (03CR) 10Legoktm: [C: 03+2] Actually run phan docker image with /cache mounted [integration/config] - 10https://gerrit.wikimedia.org/r/498295 (https://phabricator.wikimedia.org/T217479) (owner: 10Legoktm) [03:05:52] (03CR) 10Legoktm: [C: 03+2] Install php-imagick in mediawiki-phan container [integration/config] - 10https://gerrit.wikimedia.org/r/498297 (owner: 10Legoktm) [03:08:34] (03Merged) 10jenkins-bot: Actually run phan docker image with /cache mounted [integration/config] - 10https://gerrit.wikimedia.org/r/498295 (https://phabricator.wikimedia.org/T217479) (owner: 10Legoktm) [03:08:38] (03Merged) 10jenkins-bot: Install php-imagick in mediawiki-phan container [integration/config] - 10https://gerrit.wikimedia.org/r/498297 (owner: 10Legoktm) [03:08:53] !log rebuilding mediawiki-phan for https://gerrit.wikimedia.org/r/498297 [03:08:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:01:35] (03PS1) 10Legoktm: Bump mediawiki-core-php70-phan-docker to mediawiki-phan:0.1.13 [integration/config] - 10https://gerrit.wikimedia.org/r/498304 [07:46:33] RECOVERY - Host integration-slave-docker-1046 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms [07:53:27] PROBLEM - Host integration-slave-docker-1046 is DOWN: CRITICAL - Host Unreachable (172.16.1.115) [08:06:11] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:34:01] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:52:02] 10Continuous-Integration-Infrastructure, 10Growth-Team, 10StructuredDiscussions: Repeat mwext-node10-rundoc-docker failures for Flow - https://phabricator.wikimedia.org/T218978 (10MarcoAurelio) [09:06:24] (03CR) 10Zoranzoki21: [C: 03+1] "Cool!" [integration/config] - 10https://gerrit.wikimedia.org/r/498169 (owner: 10Krinkle) [09:14:58] 10Release-Engineering-Team (Kanban): Evaluate Concourse-CI - https://phabricator.wikimedia.org/T217595 (10zeljkofilipin) @LarsWirzenius is this resolved? [09:15:58] 10Release-Engineering-Team (Kanban), 10Zuul: Evaluate Zuul v3 - https://phabricator.wikimedia.org/T218138 (10zeljkofilipin) @brennen is this resolved? [09:16:16] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate GoCD - https://phabricator.wikimedia.org/T218332 (10zeljkofilipin) [09:34:16] 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions: Repeat mwext-node10-rundoc-docker failures for Flow - https://phabricator.wikimedia.org/T218978 (10MarcoAurelio) [09:35:41] 10Continuous-Integration-Config, 10Upstream: npm 6 consistently fails with "Z_DATA_ERROR: invalid distance too far back" on some repos - https://phabricator.wikimedia.org/T215562 (10MarcoAurelio) I guess this is the same as T218978 but in that case it is causing builds to fail, thus I filed a different task. F... [09:51:16] (03PS5) 10Effie Mouzeli: Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/494812 (https://phabricator.wikimedia.org/T216518) [09:54:32] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/494812 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli) [10:16:28] Project beta-scap-eqiad build #242297: 04FAILURE in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242297/ [10:27:35] Yippee, build fixed! [10:27:36] Project beta-scap-eqiad build #242298: 09FIXED in 9 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242298/ [10:33:52] (03PS1) 10Effie Mouzeli: Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) [10:33:54] (03PS1) 10Effie Mouzeli: Fix flake8 errors [tools/scap] - 10https://gerrit.wikimedia.org/r/498341 [10:37:52] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli) [10:43:38] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/498341 (owner: 10Effie Mouzeli) [10:51:22] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Read Continuous Delivery book - https://phabricator.wikimedia.org/T218984 (10zeljkofilipin) [10:51:32] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Read Continuous Delivery book - https://phabricator.wikimedia.org/T218984 (10zeljkofilipin) p:05Triage→03Normal [10:55:08] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Read Continuous Delivery book - https://phabricator.wikimedia.org/T218984 (10zeljkofilipin) [10:56:22] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Read Continuous Delivery book - https://phabricator.wikimedia.org/T218984 (10zeljkofilipin) [11:31:32] (03PS7) 10Effie Mouzeli: Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma) [11:31:34] (03PS2) 10Effie Mouzeli: Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) [11:31:36] (03PS2) 10Effie Mouzeli: Fix flake8 errors [tools/scap] - 10https://gerrit.wikimedia.org/r/498341 [11:31:38] (03PS7) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [11:33:05] 10Release-Engineering-Team (Kanban): Evaluate Concourse-CI - https://phabricator.wikimedia.org/T217595 (10LarsWirzenius) Oh, yes, I forgot to update the ticket. We discussed this within the working group and decided against Concourse. [11:33:13] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10LarsWirzenius) [11:33:16] 10Release-Engineering-Team (Kanban): Evaluate Concourse-CI - https://phabricator.wikimedia.org/T217595 (10LarsWirzenius) 05Open→03Resolved [11:33:29] (03Abandoned) 10Effie Mouzeli: Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/494812 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli) [11:36:50] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma) [11:36:52] (03CR) 10jerkins-bot: [V: 04-1] Add unit testing of scap main.py [tools/scap] - 10https://gerrit.wikimedia.org/r/492352 (owner: 10Mark Bergsma) [11:39:47] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli) [11:39:49] (03CR) 10jerkins-bot: [V: 04-1] Add more tests for main [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli) [11:42:26] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/498341 (owner: 10Effie Mouzeli) [11:43:44] (03CR) 10Mark Bergsma: [C: 04-1] "Good start, but I think test__check_sync_flag doesn't do what it's intended to do." (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/498340 (https://phabricator.wikimedia.org/T216518) (owner: 10Effie Mouzeli) [11:45:05] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [11:45:07] (03CR) 10jerkins-bot: [V: 04-1] Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [11:55:28] (03PS8) 10Effie Mouzeli: Add --canary-wait-time flag [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) [11:57:19] RECOVERY - Puppet staleness on deployment-mediawiki-09 is OK: OK: Less than 1.00% above the threshold [3600.0] [11:58:50] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [11:59:59] Hi! When I make a patch, CI doesn't run, what could have changed? [12:00:10] I'm white listed but Jenkins doesn't even vote +1 [12:00:23] Just merged this: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/SecurePoll/+/498221 and nothing is happening looking at the zuul interface [12:00:32] I think this started since yesterday evening when I made patches and they didn't trigger Jenkins [12:00:48] And I'm on the whitelist and also, when I do "recheck", nothing happens [12:00:53] Help please [12:07:30] (03CR) 10Mark Bergsma: Add --canary-wait-time flag (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [12:08:32] (03CR) 10Mark Bergsma: [C: 04-1] "I left some (design) comments on PS7 that still apply to PS8." [tools/scap] - 10https://gerrit.wikimedia.org/r/495398 (https://phabricator.wikimedia.org/T217924) (owner: 10Effie Mouzeli) [12:26:56] Hi everyone! Did something go wrong with l10n cache rebuild upon deploying the new version of MW yesterday? [12:27:55] I'm asking because on itwiki we have a message still at the old version which however updates itself upon purging the page [12:29:02] Tentatively pinging zeljkof who was in charge of the deployment [12:29:22] Daimona: there's a bug... [12:29:24] * zeljkof is looking [12:29:33] Thanks! [12:29:42] Daimona: this? https://phabricator.wikimedia.org/T218918 [12:29:51] if so, it's UBN and train blocker [12:30:01] Sounds like yes [12:30:02] Thanks [12:30:10] if you can provide more examples to it, please do [12:30:34] Yep [12:30:37] I will on phabricator [12:40:15] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate GoCD - https://phabricator.wikimedia.org/T218332 (10LarsWirzenius) [12:46:06] Project beta-scap-eqiad build #242310: 04FAILURE in 9 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242310/ [12:57:12] Yippee, build fixed! [12:57:12] Project beta-scap-eqiad build #242311: 09FIXED in 9 min 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/242311/ [13:04:47] 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions, 10Wikimedia-production-error (Shared Build Failure): Repeat mwext-node10-rundoc-docker failures for Flow - https://phabricator.wikimedia.org/T218978 (10Lucas_Werkmeister_WMDE) I’m experiencing what looks like the same issue in [If32c5... [13:04:55] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10Addshore) [13:17:17] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [13:18:37] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:51:39] 10Release-Engineering-Team (Kanban): Evaluate Argo CD - https://phabricator.wikimedia.org/T218827 (10zeljkofilipin) [13:52:26] 10Release-Engineering-Team (Kanban): Evaluate Concourse CI - https://phabricator.wikimedia.org/T217595 (10zeljkofilipin) [13:54:02] 10Release-Engineering-Team (Kanban): Evaluate GitLab CI/CD - https://phabricator.wikimedia.org/T217594 (10zeljkofilipin) [13:58:07] 10Release-Engineering-Team (Kanban): Evaluate Tekton - https://phabricator.wikimedia.org/T217912 (10zeljkofilipin) [13:59:15] 10Release-Engineering-Team (Kanban), 10Zuul: Evaluate Zuul - https://phabricator.wikimedia.org/T218138 (10zeljkofilipin) [14:24:02] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10brennen) [14:24:04] 10Release-Engineering-Team (Kanban), 10Zuul: Evaluate Zuul - https://phabricator.wikimedia.org/T218138 (10brennen) 05Open→03Resolved Yep! [15:13:39] 10MediaWiki-Codesniffer, 10MediaWiki-Documentation, 10Documentation: Doxygen doesn't handle `@inheritDoc` by default, only `@inheritdoc` - https://phabricator.wikimedia.org/T219001 (10Anomie) [15:17:42] 10MediaWiki-Codesniffer, 10MediaWiki-Documentation, 10Documentation: Doxygen doesn't handle `@inheritDoc` by default, only `@inheritdoc` - https://phabricator.wikimedia.org/T219001 (10Anomie) The simplest upstream fix, if they don't want it to be specific to PHP, would be to just add the mixed-case entry alo... [15:23:30] Anything seeming to be going on with mwext-node10-rundoc-docker? [15:23:50] In wikibase land we're seeing lots of failures [15:24:31] Not sure if I understand jenkins-blue fully but in general it looks very red recently: https://integration.wikimedia.org/ci/blue/organizations/jenkins/mwext-node10-rundoc-docker/activity [15:29:55] * paladox is gong to work on getting the change for https://phabricator.wikimedia.org/T218739 merged today && add support for gerrit 2.16/2.15 to it! [15:51:09] PROBLEM - Puppet staleness on deployment-db04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:51:45] sweet [15:54:28] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [3.0] [16:06:02] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10phan: mwext-php70-phan-docker doesn't use composer cache - https://phabricator.wikimedia.org/T217479 (10Umherirrender) It seems the cache is now filled - on https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/24031/consoleFull there... [16:28:03] PROBLEM - Host integration-publishing02 is DOWN: CRITICAL - Host Unreachable (172.16.4.5) [16:28:58] James_F: what's the task tracking the "npm ERR! invalid distance too far back" you and Krink.le were looking into last night? cc tarrow [16:29:49] greg-g: Don't want to speak too soon but seems to have gotten sorted [16:31:50] greg-g: We 'fixed' the issue for VE by reverting the job to the npm6 runner, but that's a short term solution. [16:32:04] (Node6 goes EOL at the end of next month.) [16:33:10] It's apparently an issue between the version of zlib in Debian and node's internal handling of stuff. What fun. [16:51:27] 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions, 10Wikimedia-production-error (Shared Build Failure): Repeat mwext-node10-rundoc-docker failures for Flow - https://phabricator.wikimedia.org/T218978 (10Lucas_Werkmeister_WMDE) This no longer seems to be happening now. [16:59:01] 10Release-Engineering-Team (Next), 10ChangeProp, 10Release Pipeline, 10serviceops, and 2 others: Migrate changeprop to kubernetes - https://phabricator.wikimedia.org/T213193 (10Ottomata) BTW, I'm currently having a node-rdkafka + k8s related issue in EventGate: {T218268}. It is possible that there is some... [17:12:26] Anyone knows which credentials we're supposed to use to access logstash-beta? My wikitech account doesn't work. [17:14:39] stephanebisson: according to my Firefox, the site says: [17:14:41] Logstash (ssh deployment-deploy01.deployment-prep.eqiad.wmflabs sudo cat /root/secrets.txt) [17:14:54] but IIRC some browsers (Safari?) don’t display that warning [17:14:59] s/warning/message/ [17:15:23] Lucas_WMDE: Thanks! Chrome doesn't show that :( [17:15:29] :( [17:20:44] Is automatic config deployment to labs working on Fridays? [17:27:02] stephanebisson: I'll repeat what Krenair told me when I tried to access logstash-beta with my wikitech credentials: change your password, it could be compromised ;-) [17:28:13] yeah never show your LDAP password to a non-production system [17:28:20] especially if you are in one of the NDA groups [17:28:37] also, don't use browsers which don't properly show the message we send [17:29:40] stephanebisson, the automatic deployment in beta should still run on fridays AFAIK [17:30:39] 10Phabricator (2019-03-20): Moving tasks between columns on workboard changed task priority (when sorting tasks by priority) - https://phabricator.wikimedia.org/T136400 (10Aklapper) 05Open→03Resolved This cannot happen anymore as workboard columns in the Wikimedia Phabricator instance now have grouping, by d... [17:31:28] stephanebisson: https://integration.wikimedia.org/ci/view/Beta/ says beta-code-update-eqiad and beta-scap-eqiad ran successfully some minutes ago [17:32:57] Right, I'm puzzled by this change: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/498416/ [17:33:23] It doesn't seem to be there and gerrit doesn't have the confirmation message that it has been synced [17:35:29] looks like it's not working properly [17:35:39] /27/8 [17:35:46] deployment-deploy01:/srv/mediawiki-staging HEAD is f88f50dd0e51e3df19cfb7fd3fb8e527673f071b [17:35:52] Fri Mar 15 09:29:06 2019 +0000 [17:40:19] Who usually fixes those jobs? [17:40:37] Should I create a phab task? [17:41:56] looks like https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/ hasn't run [17:42:38] sounds like a jenkins admin should take a look [17:44:04] hashar might know? [17:45:51] <28stephanebisson18> Should I create a phab task? [17:45:53] yes please [17:45:56] always fill a task :) [17:45:59] they are cheap! [17:46:19] ok [17:47:20] seems like https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/ is no more triggered by zuul bah [17:47:57] yeah, 7 days ago, why? [17:49:07] 10Release-Engineering-Team: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10SBisson) [17:51:22] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) p:05Triage→03Unbreak! [17:51:52] 10Release-Engineering-Team (Kanban): Evaluate Argo Workflow - https://phabricator.wikimedia.org/T218827 (10dduvall) [17:56:45] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) [17:56:48] 10Phabricator (Upstream), 10Upstream: [Send Message] button on person page is shown to users without permission, showing error message - https://phabricator.wikimedia.org/T91438 (10epriestley) This is likely obsolete because the "Send Message" action was removed a long time ago. I don't have any current plans... [18:02:13] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) The job is triggered when a change is merged in operations/mediawiki-config. Commits that have been made: f21763295 Mon Mar 18 07:58:... [18:10:30] stephanebisson: Krenair I have no idea what is going on :((( [18:25:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) For https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/498416/ from Zuul debug logs: It gets the `change-merged` event:... [18:25:33] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments, 10User-zeljkofilipin: 1.33.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T206676 (10Jdforrester-WMF) [18:34:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) I have absolutely no idea what is going on. Meanwhile I have manually enqueued the latest change to mediawiki-config. On contint1001.wi... [18:34:36] stephanebisson: Krenair I have added the job manually for now [18:45:03] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10VisualEditor, 10User-Ryasmeen: CI outage for mediawiki/extensions/VisualEditor – job "mwext-node10-rundoc-docker" always fails and we can't merge code due to repo configuration - https://phabricator.wikimedia.org/T218963 (10mat... [18:45:52] Hi, I asked earlier today about my Gerrit account not triggering CI [18:46:07] When I make patches, nothing happens, could someone have a look please? [18:48:23] xSavitar what email do you have set as primary? [18:48:44] Ohhhhhh! [18:48:52] paladox, you've answered my question already [18:48:58] How did I not think of that? :( [18:49:03] yeh [18:49:05] paladox: It's the new email I added [18:49:08] yup [18:49:43] And I'm trying to separate accounts so I don't want work stuff into @gmail.com anymore [18:49:51] I want to separate work things from real life stuff [18:50:07] paladox: Can I add the new one to CI and you review? [18:50:52] xSavitar i have no +2 but hashar may be able to [18:51:43] might be gone at this hour (friday evening) [18:52:27] Okay, I'll just revert for now and use the old email until I get the new one into zuul [18:52:38] paladox: I owe you a beer :) [18:52:47] heh [18:53:10] Seriously, I didn't think of that [18:53:27] I honestly forgot of the whitelist that it's tied to emails [18:53:34] Thanks again paladox :), much appreciated [19:05:25] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10dduvall) [19:05:31] 10Release-Engineering-Team (Kanban): Evaluate Argo Workflow - https://phabricator.wikimedia.org/T218827 (10dduvall) 05Open→03Resolved First, some clarification about the various Argo projects. From IRC (2019-03-21 UTC-7): ` 2:59 PM Dan Duvall argo projects are starting to make sense finally 3:0... [19:14:11] (03PS1) 10Hashar: test postmerge rejects l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/498453 (https://phabricator.wikimedia.org/T219017) [19:14:12] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) Still from Zuul debug logs, it no more matches events since Mar 15th indeed :( ` $ grep -c 'Event change-merged operations/mediawiki-c... [19:15:39] (03CR) 10jerkins-bot: [V: 04-1] test postmerge rejects l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/498453 (https://phabricator.wikimedia.org/T219017) (owner: 10Hashar) [19:16:18] (03CR) 10Thcipriani: [C: 03+2] Remove requirement for properties file, import coverage if present [integration/config] - 10https://gerrit.wikimedia.org/r/494548 (https://phabricator.wikimedia.org/T215176) (owner: 10Kosta Harlan) [19:16:34] (03PS2) 10Hashar: test postmerge rejects l10n-bot [integration/config] - 10https://gerrit.wikimedia.org/r/498453 (https://phabricator.wikimedia.org/T219017) [19:17:07] 10Continuous-Integration-Config, 10Growth-Team, 10StructuredDiscussions, 10Wikimedia-production-error (Shared Build Failure): Repeat mwext-node10-rundoc-docker failures for Flow - https://phabricator.wikimedia.org/T218978 (10Lucas_Werkmeister_WMDE) p:05Triage→03Unbreak! Okay, here are some more failure... [19:19:11] (03Merged) 10jenkins-bot: Remove requirement for properties file, import coverage if present [integration/config] - 10https://gerrit.wikimedia.org/r/494548 (https://phabricator.wikimedia.org/T215176) (owner: 10Kosta Harlan) [19:24:39] !log clean old docker-pkg image from contint1001 [19:24:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:27:34] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate GoCD - https://phabricator.wikimedia.org/T218332 (10LarsWirzenius) Looked at GoCD, by Thoughtworks. A Java application, divided into a server, which also provides a web UI, and an an agent, which needs to be run on each worker. Similar to trad... [19:27:41] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Evaluate GoCD - https://phabricator.wikimedia.org/T218332 (10LarsWirzenius) 05Open→03Resolved a:03LarsWirzenius [19:27:43] 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Consider and evaluate possible new CI tooling - https://phabricator.wikimedia.org/T217325 (10LarsWirzenius) [19:34:49] !log updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/494548/ [19:34:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:47:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) I am looking at projects having a `postmerge` pipeline and try to find the last time an event got accepted. media... [19:47:48] ar [19:47:53] so hmm postmerge is broken [19:48:13] seem ssomething happened between 2019-03-15 21:46 and 22:03 [19:51:42] (03PS1) 10Hashar: Revert "Disable post-merge doc/coverage publish for l10n commits" [integration/config] - 10https://gerrit.wikimedia.org/r/498466 (https://phabricator.wikimedia.org/T219017) [19:51:49] (03PS2) 10Hashar: Revert "Disable post-merge doc/coverage publish for l10n commits" [integration/config] - 10https://gerrit.wikimedia.org/r/498466 (https://phabricator.wikimedia.org/T219017) [19:51:55] pretty sure that is the one [19:57:00] (03CR) 10Hashar: [C: 03+2] "I have absolutely no idea why it would break postmerge, but it seems to do somehow :(((((((" [integration/config] - 10https://gerrit.wikimedia.org/r/498466 (https://phabricator.wikimedia.org/T219017) (owner: 10Hashar) [19:58:30] (03Merged) 10jenkins-bot: Revert "Disable post-merge doc/coverage publish for l10n commits" [integration/config] - 10https://gerrit.wikimedia.org/r/498466 (https://phabricator.wikimedia.org/T219017) (owner: 10Hashar) [19:59:13] 10MediaWiki-Codesniffer: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants null even if type is nullable - https://phabricator.wikimedia.org/T218816 (10Legoktm) >>! In T218816#5041373, @Anomie wrote: > `?string` and `string|null` mean the same thing, I suppose the question is which of those we want... [19:59:45] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/498466/ # T219017 [19:59:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:59:47] T219017: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 [20:02:50] (03PS3) 10Hashar: Disable post-merge doc/coverage publish for l10n commits [integration/config] - 10https://gerrit.wikimedia.org/r/498453 [20:03:13] (03PS6) 10Paladox: Docker: Add bazel image [integration/config] - 10https://gerrit.wikimedia.org/r/493638 [20:03:15] (03PS24) 10Paladox: Gerrit: Add CI for operations/software/gerrit (includes new docker image) [integration/config] - 10https://gerrit.wikimedia.org/r/493328 [20:03:25] (03CR) 10Hashar: "Uploaded again as https://gerrit.wikimedia.org/r/#/c/integration/config/+/498453/" [integration/config] - 10https://gerrit.wikimedia.org/r/496976 (owner: 10Krinkle) [20:04:06] (03CR) 10Hashar: [C: 04-2] "The code is the same, I have just added some tests which pass and thus do not highlight the issue on T219017 :(" [integration/config] - 10https://gerrit.wikimedia.org/r/498453 (owner: 10Hashar) [20:05:56] (03PS1) 10Legoktm: Treat "mixed" as already including null [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/498472 (https://phabricator.wikimedia.org/T218324) [20:06:15] 10MediaWiki-Codesniffer, 10Patch-For-Review: MediaWiki.Commenting.FunctionComment.DefaultNullTypeParam wants redundant "mixed|null" - https://phabricator.wikimedia.org/T218324 (10Legoktm) a:03Legoktm [20:07:57] 10Release-Engineering-Team (Kanban): Evaluate Argo - https://phabricator.wikimedia.org/T218827 (10dduvall) [20:08:21] 10Continuous-Integration-Config: post-merge jenkins job not run after merge for maven based search projects - https://phabricator.wikimedia.org/T218550 (10hashar) That got raised today as well for operations/mediawiki-config , I have no idea about the root cause though :-/ I guess I should have investigated ear... [20:08:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) [20:08:36] 10Continuous-Integration-Config: post-merge jenkins job not run after merge for maven based search projects - https://phabricator.wikimedia.org/T218550 (10hashar) [20:09:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) @Gehel noticed the same issue on `search/glent` a few days ago and I did some investigation at the time T218550#5... [20:10:17] 10Continuous-Integration-Config: post-merge jenkins job not run after merge for maven based search projects - https://phabricator.wikimedia.org/T218550 (10hashar) > I have added it manually with `zuul enqueue` CLI utility on contint1001 Note, when manually adding changes with `zuul enqueue` the event filters ar... [20:18:56] (03CR) 10Umherirrender: "Maybe autofix mixed|null back to mixed?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/498472 (https://phabricator.wikimedia.org/T218324) (owner: 10Legoktm) [20:23:29] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer: Add cache support to PHPCode_Sniffer jobs - https://phabricator.wikimedia.org/T146644 (10Umherirrender) >>! In T146644#4683761, @Legoktm wrote: > There's a second (and probably more important) bug where `vendor/mediawiki/mediawiki-codesniffer/MediaWiki/... [20:25:20] 10Gerrit, 10Release-Engineering-Team, 10Operations, 10serviceops: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10Paladox) [20:25:57] (03CR) 10Legoktm: [C: 04-1] "Yep... https://codesearch.wmflabs.org/search/?q=mixed%5C%7Cnull&i=nope&files=&repos=" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/498472 (https://phabricator.wikimedia.org/T218324) (owner: 10Legoktm) [21:05:25] PROBLEM - Host deployment-sessionstore01 is DOWN: CRITICAL - Host Unreachable (172.16.3.4) [21:06:44] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI job beta-mediawiki-config-update-eqiad has stopped running - https://phabricator.wikimedia.org/T219017 (10hashar) 05Open→03Resolved a:03hashar It is working again, albeit l10n-bot enqueues jobs in `postmerge` pipeline. [21:40:29] (03Abandoned) 10Paladox: Fix blocking users [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/497429 (owner: 10Paladox) [22:35:57] what's wrong with https://integration.wikimedia.org/ci/job/mwext-node10-rundoc-docker/113/console [22:36:12] passes on 'test', fails on 'gate-and-submit' [22:36:21] or doesn't pass anywhere [22:36:26] it's annoying [22:36:48] James_F: ^ what should I tell people? Sorry, I haven't had a chance to review phab tasks yet [22:37:17] it's for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Flow/+/498213/ [22:37:45] I don't want to override jenkins even if this is not a code change [23:00:38] * paladox is feeling excited that his change is getting merged upstream to support the new session plugin! [23:03:16] paladox: Nice! [23:04:08] yup [23:04:15] nice work, paladox [23:04:21] thanks :) [23:17:50] 10Gerrit, 10Release-Engineering-Team (Kanban): Bless Brennen with Gerrit administrator rights - https://phabricator.wikimedia.org/T218858 (10Paladox) +1 [23:18:26] 10Continuous-Integration-Infrastructure: debian-glue does not support autopkgtest version 5 - https://phabricator.wikimedia.org/T212773 (10hashar) [23:18:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Packaging, 10Upstream: jenkins-debian-glue should run the lintian version from cowbuilder instead of from host - https://phabricator.wikimedia.org/T186494 (10hashar) [23:18:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, 10Patch-For-Review: Upgrade jenkins-debian-glue to v0.20.0 - https://phabricator.wikimedia.org/T212774 (10hashar) 05Open→03Resolved Seems good so far. Thank you very much. I will later look at mig... [23:19:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: debian-glue errors out with: hooks/B90lintian: line 21: realpath: command not found - https://phabricator.wikimedia.org/T193906 (10hashar) [23:19:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Packaging, 10Upstream: jenkins-debian-glue should run the lintian version from cowbuilder instead of from host - https://phabricator.wikimedia.org/T186494 (10hashar) 05Stalled→03Resolved It took a while to get it released b... [23:22:41] (03Abandoned) 10Hashar: jenkins-debian-glue: Skip autopkgtests [integration/config] - 10https://gerrit.wikimedia.org/r/463507 (owner: 10Legoktm) [23:23:19] 10Continuous-Integration-Infrastructure: debian-glue does not support autopkgtest version 5 - https://phabricator.wikimedia.org/T212773 (10hashar) 05Open→03Resolved a:03hashar I have upgraded to jenkins-debian-glue v0.20.0 [23:28:13] 10Gerrit, 10Release-Engineering-Team, 10Operations: Create Gerrit Administrator right policy - https://phabricator.wikimedia.org/T218686 (10Peachey88) [23:29:41] I have a little beta puppet issue; $someone changed the puppetmaster for one of my instances and I've just discovered it and changed it back. during that interlude puppet has been broken, but the main thing is... [23:30:12] that I cleaned out the cert for my instance on beta's puppetmaster (-03) [23:30:34] and cleaned up the ssl dir on the instance but beta's puppetmaster keeps serving me the same (deleted) cert [23:30:41] which the instance can't use [23:31:04] did you clean both the puppetmaster and puppet host? [23:31:06] any ideas? I'm hesitant to restart the master [23:31:15] I removed the ssl dir on the instance [23:31:27] I cleaned the cert and checked it was gone from the relevant dirs, on the puppetmaster [23:31:45] and yet the puppetmaster is still able to serve that missing cert [23:32:14] you were looking in /var/lib/puppet/ssl right? [23:32:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10VisualEditor, 10User-Ryasmeen: CI outage for mediawiki/extensions/VisualEditor – job "mwext-node10-rundoc-docker" always fails and we can't merge code due to repo configuration - https://phabricator.wikimedia.org/T218963 (10has... [23:32:55] you want /var/lib/puppet/server/ssl [23:33:15] on the master I have [23:33:18] find /var/lib/puppet/ -name deployment-dumps-puppetmaster\* [23:33:21] nothing is there [23:33:43] fles were removed from [23:33:46] /var/lib/puppet/server/ssl/..... [23:33:51] on the master [23:34:13] which host was it again that used dumps-puppetmaster? [23:34:15] snapshot something? [23:34:56] uh huh but ignore that [23:35:03] I'm going to have to patch all that up after ths [23:35:20] which instance is supposed to be talking to which puppetmaster and what is it talking to instead? [23:35:56] deployment-dumps-puppetmaster02 was supposed to do what it always did and talk to [23:36:07] deployment-puppetmaster03.deployment-prep.eqiad.wmflabs [23:36:28] but on jan 25 apparently the config files were rewrritten (and it seems the hiera setting changed or removed) [23:36:34] ugh [23:36:41] so it talked to labs-puppetmaster instead [23:36:46] okay [23:36:48] one sec [23:36:49] I have edited that in the conf files [23:37:02] (well, put the old filebucket conf files back) [23:38:33] there's no logs so I have no idea how that key got altered, and that day I think I was stuck in the airport in Turkey for 24 hours so I have 0 info [23:40:06] * Krenair is trying to get it to talk to our puppetmaster again [23:40:23] thanks. I guess you are getting the same whine about the cert? [23:40:58] Certificate fingerprint: 09:18:46:84:57:4C:34:24:00:12:15:AE:B0:46:68:14:0A:DA:85:6D:BC:F2:A7:ED:72:D8:6C:5E:3D:5A:FA:E4 [23:41:00] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [23:41:27] that's the one it told me initially, so I went and cleaned everything up. and I know the cert is gone off the beta puppetmaster, so how is it providing it [23:41:33] unless there's a cache.. ? [23:43:33] looks like this thing hasn't successfully run puppet for a while [23:44:18] apergos, okay I think dumps-puppetmaster02 is now configured properly and happy again? [23:44:40] yes [23:44:47] can you let me know what you did? [23:44:58] I don't want to have to nag you the next time,nor lose a bunch of time either [23:45:28] well after a few false starts, basically [23:45:52] * fixed /etc/puppet/puppet.conf to point to puppetmaster03 instead of labs-puppetmaster [23:46:08] * moved /var/lib/puppet/ssl out of the way [23:46:13] ah you know what, I probably only updated the conf.d pieces [23:46:27] * cleaned old cert off puppetmaster03 [23:46:32] * ran puppet agent on dumps-puppetmaster [23:46:37] * signed new cert on puppetmaster03 [23:46:37] uh, I had already cleaned it off [23:46:41] honest [23:46:42] * ran puppet agent on dumps-puppetmaster again [23:46:51] okay :) [23:46:59] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Upgrade Gerrit to 2.15.12 - https://phabricator.wikimedia.org/T218515 (10Paladox) [23:47:00] dd it say it was removing the cert and give paths, when you cleaned it off? [23:47:06] or just 'revoking cert 84' ? [23:47:14] it just said revoking [23:47:20] ok, so it was already gone then [23:47:21] so I think it had already been done as you said [23:47:22] yeah [23:47:39] I bet it was my not thinking of needing to edit the assembled puppet conf file [23:47:57] because of course, the exec that assembles the pieces in puppet.conf.d doesn't run.... [23:48:35] thanks for catching that, I sure wish I knew how the hiera key was changed/removed, but there's no logs for edits via the interface, i checked all sal, this channel's log (I was getting desperate), [23:48:37] yeah... I'm still not entirely convinced it's regenerated puppet.conf based on the conf.d files [23:48:42] git log for puppet repo... [23:48:45] all that [23:49:05] 10Continuous-Integration-Config, 10Gerrit: Run `git submodule sync` when updating extensions.git clones - https://phabricator.wikimedia.org/T218929 (10hashar) 05Open→03Declined Nop CI does not use mediawiki/extensions.git . It is unclear to me what the issue is about. Feel free to reopen with more details... [23:49:07] 10Gerrit, 10Wikimedia-General-or-Unknown, 10Documentation, 10Epic, and 2 others: Update Gerrit /r/p/ links to /r/ - https://phabricator.wikimedia.org/T218844 (10hashar) [23:49:07] and there was no trace, just 'jan 25 01:14' new fles written [23:49:13] anyways, thanks for the fix [23:49:28] now i can go whack snapshot instance into shape and then be ready for testing this weekend [23:50:12] I take it the separate dumps-puppetmaster is still necessary [23:51:13] yep [23:51:18] ok [23:51:22] np [23:51:35] good luck with the testing [23:51:48] thank you! [23:52:31] did you already clean up snapshot01? [23:52:46] because puppet runs over there and I figured it would whine about its master having a new cert etc [23:53:55] well, it isn't and that's cool [23:54:07] I actually didn't touch it beyond running puppet [23:54:10] which didn't do much [23:54:11] * apergos wanders off, it's about 2 am so, tme to break for the night [23:54:23] okey dokey! [23:54:24] I think the master cert used on dumps-puppetmaster didn't change or something, not sure [23:54:37] I'll look at it later sometmie if I need [23:54:41] ... a headache :-D [23:54:42] /var/lib/puppet/server/ssl is separate [23:54:44] thanks again [23:54:49] hopefully when the puppetmaster restarts it doesn't break everything [23:55:05] if it does, well, we figured out how to set this up once, I'm sure we can re-do it [23:55:11] yup