[02:15:48] PROBLEM - Host deployment-redis02 is DOWN: CRITICAL - Host Unreachable (10.68.16.231) [02:15:52] PROBLEM - Host deployment-redis01 is DOWN: CRITICAL - Host Unreachable (10.68.16.177) [02:18:00] PROBLEM - Host deployment-dumps-puppetmaster is DOWN: CRITICAL - Host Unreachable (10.68.21.153) [02:24:21] PROBLEM - Host deployment-puppetmaster02 is DOWN: CRITICAL - Host Unreachable (10.68.21.200) [02:36:45] Project beta-scap-eqiad build #212707: 04FAILURE in 3 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/212707/ [02:47:40] Yippee, build fixed! [02:47:41] Project beta-scap-eqiad build #212708: 09FIXED in 3 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/212708/ [05:16:37] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Stop using Differential for code review - https://phabricator.wikimedia.org/T191182#4304358 (10Dzahn) ``` mysql:phuser@m3-slave.eqiad.wmnet [(none)]> SELECT r.name FROM phabricator_repository.repository r JOIN phabricator_differential.differentia... [08:01:59] 10Phabricator, 10Project-Admins: Creating private tasks by filling a web form (for AffCom) - https://phabricator.wikimedia.org/T89335#4304432 (10Aklapper) 05stalled>03declined >>! In T89335#4189762, @Aklapper wrote: > @Qgil / @Varnent: If this is still wanted, can you provide these details please? No repl... [08:07:10] 10Gerrit: Gerrit cannot find I20dbace60e8d5b95e858e00595809171dba4c555 (from 2012) - https://phabricator.wikimedia.org/T197832#4304442 (10hashar) 05Open>03Resolved a:03hashar Getting all references from Gerrit (including `refs/changes/*` and `refs/notes/*` ``` git clone --bare https://gerrit.wikimedia.org/... [08:27:20] 10Release-Engineering-Team (Someday), 10MediaWiki-Release-Tools: make-release doesn't list mediawiki-core - https://phabricator.wikimedia.org/T180540#4304468 (10Aklapper) >>! In T180540#4206865, @demon wrote: > I think I fixed this? Looks like: https://lists.wikimedia.org/pipermail/wikitech-l/2018-June/090200... [09:07:51] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Stop using Differential for code review - https://phabricator.wikimedia.org/T191182#4304526 (10MarcoAurelio) As for Scap there's {T191373}. As for the others, I haven't looked if there's a proposed migration task. [09:11:15] (03PS1) 10Giuseppe Lavagetto: Add isc-dhcp-server to the installed packages [integration/config] - 10https://gerrit.wikimedia.org/r/441349 [09:11:40] <_joe_> heh, better ask here [09:11:52] <_joe_> hashar: if you're around, I would really need a +2 to that patch [09:11:56] <_joe_> and a build :P [10:00:03] (03PS1) 10MarcoAurelio: Archive the CommunityVoice extension [integration/config] - 10https://gerrit.wikimedia.org/r/441358 (https://phabricator.wikimedia.org/T196618) [10:01:13] _joe_: sorry I was / still in audio with Zeljko [10:01:29] _joe_: I guess you want to write a test that use the dhcp binary to validate the dhcp config files are correct ? :) [10:09:07] (03PS2) 10MarcoAurelio: Mark repository as read only [extensions/CommunityVoice] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/441361 (https://phabricator.wikimedia.org/T196618) [10:10:21] (03PS3) 10MarcoAurelio: Mark repository as read only [extensions/CommunityVoice] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/441361 (https://phabricator.wikimedia.org/T196618) [10:14:28] antoine: https://github.com/wikimedia/mediawiki-extensions-CustomPage [10:14:35] ehm hashar ^^ [10:14:45] éffacer s.v.p [10:19:16] Hauskatze: ack :] [10:19:32] Hauskatze: and again thank you for all the tasks you are doing on #cleanup .You cant imagine how hlepful it is [10:19:55] Well, as weird as it might sound, I actually like doing that stuff [10:20:01] I'm glad it's useful [10:20:03] Hauskatze: done [10:20:14] hashar: I think I have another mirror to delete [10:21:03] https://github.com/wikimedia/mediawiki-extensions-CommunityVoice <-- although this one is not yet archived in Gerrit the patches are already there [10:21:26] we can wait or delete it now, as you prefer [10:31:54] Hauskatze: I think Gerrit will end up recreating it automagically when the patch to add OBSOLETE file is merged [10:32:04] though I am not sure whether handle that automatically [10:33:03] okay so let's wait until everything is setup in gerrit [10:33:19] no_justification may know better though, I'm not sure how GitHub mirrors Gerrit [10:33:27] (03PS2) 10Hashar: Add isc-dhcp-server to the installed packages [integration/config] - 10https://gerrit.wikimedia.org/r/441349 (owner: 10Giuseppe Lavagetto) [10:33:58] Hauskatze: so in doubt, I just delete the github repos at the end of the process :] [10:34:40] dies kat agreez [10:35:06] (03CR) 10Hashar: "Edited commit message with "Intended to lint DHCP configuration files." based on pure guessing." [integration/config] - 10https://gerrit.wikimedia.org/r/441349 (owner: 10Giuseppe Lavagetto) [10:35:08] (03CR) 10Hashar: [C: 032] Add isc-dhcp-server to the installed packages [integration/config] - 10https://gerrit.wikimedia.org/r/441349 (owner: 10Giuseppe Lavagetto) [10:37:16] (03Merged) 10jenkins-bot: Add isc-dhcp-server to the installed packages [integration/config] - 10https://gerrit.wikimedia.org/r/441349 (owner: 10Giuseppe Lavagetto) [10:37:43] <_joe_> hashar: yes you're correct [10:37:52] <_joe_> sorry, I was working on something else now [10:40:08] (03PS1) 10Hashar: Bump job to use operations-puppet:0.3.2 [integration/config] - 10https://gerrit.wikimedia.org/r/441367 [10:40:13] !log Buidling docker image releng/operations-puppet:0.3.2 https://gerrit.wikimedia.org/r/c/integration/config/+/441349/ [10:40:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:40:34] _joe_: no worries, took me time to notice your ping since I was in a google hangouts [10:40:51] image is rebuilding and there is a pending patch to switch the job to the new image [10:41:52] <_joe_> <3 [11:06:45] _joe_: build and uploaded to registry [11:07:41] (03CR) 10Hashar: [C: 032] Bump job to use operations-puppet:0.3.2 [integration/config] - 10https://gerrit.wikimedia.org/r/441367 (owner: 10Hashar) [11:07:48] <_joe_> thanks a lot [11:08:00] and job refreshed [11:08:49] (03CR) 10Hashar: [C: 032] "Deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/441367 (owner: 10Hashar) [11:09:53] (03Merged) 10jenkins-bot: Bump job to use operations-puppet:0.3.2 [integration/config] - 10https://gerrit.wikimedia.org/r/441367 (owner: 10Hashar) [11:35:48] (03CR) 10Zfilipin: Really make WikibaseLexeme node selenium job voting (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/441178 (owner: 10WMDE-leszek) [11:39:08] (03CR) 10Zfilipin: [C: 031] Remove no longer used node selenium + composer job [integration/config] - 10https://gerrit.wikimedia.org/r/441182 (owner: 10WMDE-leszek) [11:39:35] (03CR) 10Zfilipin: [C: 031] Remove redundant WikibaseLexeme jobs [integration/config] - 10https://gerrit.wikimedia.org/r/441181 (owner: 10WMDE-leszek) [12:02:58] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568#4305040 (10Dzahn) a:05Dzahn>03None Meanwhile we have wmf4727 and it is in site.pp and using the phabricator puppet... [12:04:21] !log github: deleting https://github.com/wikimedia/mediawiki-extensions-EtherEditor | T197698 [12:04:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:04:24] T197698: Archive the EtherEditor extension - https://phabricator.wikimedia.org/T197698 [12:11:09] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568#4305058 (10Paladox) Is the repos being resynced to phab1002? [12:14:29] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568#4305064 (10Paladox) Path to be rsync is /srv/repos [12:16:45] 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568#4305066 (10Dzahn) The code is there to allow a user to do it. But it's not auto-syncing in the background. It needs a... [12:22:22] 10Phabricator, 10Project-Admins, 10Epic: Replace all "tracking" tasks with tags/projects (if unbounded), relabel them to outcome tasks (if bounded), or kill them entirely (if pointless) - https://phabricator.wikimedia.org/T192655#4305081 (10Aklapper) [12:27:13] 10Phabricator: Remove approval requirement for new accounts, or patch everything in Phabricator to allow unapproved users to be treated as logged out for permissions purposes - https://phabricator.wikimedia.org/T197550#4305085 (10Aklapper) https://secure.phabricator.com/T13154#239449 offers code that we could co... [12:31:23] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice: BlueSpiceFoundation REL1_27 cause DatabaseUpdater::setAppliedUpdates Error: 1062 Duplicate entry 'updatelist-1.27.4-15295832440' - https://phabricator.wikimedia.org/T197861#4305089 (10hashar) p:05Triage>03Nor... [12:31:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4305103 (10hashar) The job now passes on master with php7.0 / mysql. A deprecation warning was blocking the... [12:34:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice: BlueSpiceFoundation REL1_27 cause DatabaseUpdater::setAppliedUpdates Error: 1062 Duplicate entry 'updatelist-1.27.4-15295832440' - https://phabricator.wikimedia.org/T197861#4305105 (10hashar) [12:38:25] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice: BlueSpiceFoundation REL1_27 cause DatabaseUpdater::setAppliedUpdates Error: 1062 Duplicate entry 'updatelist-1.27.4-15295832440' - https://phabricator.wikimedia.org/T197861#4305123 (10hashar) [12:53:29] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice: BlueSpiceFoundation REL1_27 cause DatabaseUpdater::setAppliedUpdates Error: 1062 Duplicate entry 'updatelist-1.27.4-15295832440' - https://phabricator.wikimedia.org/T197861#4305139 (10hashar) The duplicate key f... [12:58:27] (03PS1) 10Hashar: Switch BlueSpiceFoundation to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441386 (https://phabricator.wikimedia.org/T183512) [12:59:38] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4305147 (10hashar) [13:00:02] (03CR) 10Hashar: [C: 032] Switch BlueSpiceFoundation to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441386 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [13:01:35] (03Merged) 10jenkins-bot: Switch BlueSpiceFoundation to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441386 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [13:17:38] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4305154 (10hashar) BlueSpiceFoundation is migrated \o/ Validated usin... [13:18:23] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4305155 (10hashar) I have migrated BlueSpiceFoundation to use the new #Quibble Jenkins jobs. Validated usin... [13:20:01] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4305156 (10hashar) [13:24:14] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4305172 (10hashar) [13:37:53] 10Phabricator: Remove approval requirement for new accounts, or patch everything in Phabricator to allow unapproved users to be treated as logged out for permissions purposes - https://phabricator.wikimedia.org/T197550#4305216 (10mmodell) Also: >>! In https://secure.phabricator.com/T13154#239449, **epriestley*... [13:44:35] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4305239 (10hashar) [13:53:30] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4305257 (10hashar) I am running a script on the CI server (contint1001) which runs the job `quibble-compose... [13:59:43] (03PS2) 10Rafidaslam: __init__.py: Supress unintentional 'true' print in `ansi_codes()` [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/438234 (https://phabricator.wikimedia.org/T195078) [14:00:51] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#4305263 (10Osnard) [14:00:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4305264 (10Osnard) [14:07:17] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4305306 (10hashar) [14:22:59] 10Gerrit, 10Developer-Relations, 10Documentation: Update Wikimedia's Gerrit documentation for new user interface in Gerrit 2.15 - https://phabricator.wikimedia.org/T179759#4305350 (10Paladox) The WIP documentation for polygerrit needs documenting as this would have resolved T197238 (IE the "Start Review" but... [15:39:40] (03CR) 10WMDE-leszek: Really make WikibaseLexeme node selenium job voting (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/441178 (owner: 10WMDE-leszek) [15:46:01] Hi, I'm finding that a branch (and it's commits) aren't appearing in phabricator but they are present in the github repo phabricator is "observing". Specifically: 'fingerprint-entity' from https://github.com/wmde/WikibaseDataModelJavaScript.git to 'wikibase-data-model' on phabricator. Where should I be looking to figure out what is going on? [15:47:33] 10Release-Engineering-Team (Kanban), 10Wikidata, 10User-zeljkofilipin: jenkins_jobs.errors.JenkinsJobsException: Duplicate entry found in '/src/integration/config/jjb/selenium.yaml: 'WikibaseLexeme' already defined - https://phabricator.wikimedia.org/T197882#4305615 (10zeljkofilipin) [15:47:51] 10Release-Engineering-Team (Kanban), 10Wikidata, 10User-zeljkofilipin: jenkins_jobs.errors.JenkinsJobsException: Duplicate entry found in '/src/integration/config/jjb/selenium.yaml: 'WikibaseLexeme' already defined - https://phabricator.wikimedia.org/T197882#4305626 (10zeljkofilipin) [15:48:21] 10Release-Engineering-Team (Kanban), 10Wikidata, 10User-zeljkofilipin: jenkins_jobs.errors.JenkinsJobsException: Duplicate entry found in '/src/integration/config/jjb/selenium.yaml: 'WikibaseLexeme' already defined - https://phabricator.wikimedia.org/T197882#4305615 (10zeljkofilipin) p:05Triage>03Normal [15:49:15] tarrow: link to phab repo for my convenience? I'm in a meeting :) [15:50:44] greg-g: Thanks https://phabricator.wikimedia.org/source/wikibase-data-model/ . [15:50:46] 10Release-Engineering-Team (Kanban), 10Wikidata, 10User-zeljkofilipin: jenkins_jobs.errors.JenkinsJobsException: Duplicate entry found in '/src/integration/config/jjb/selenium.yaml: 'WikibaseLexeme' already defined - https://phabricator.wikimedia.org/T197882#4305637 (10zeljkofilipin) [15:51:22] * Hauskatze looks [15:51:46] looks like maybe we are cause by https://secure.phabricator.com/book/phabricator/article/diffusion_updates/ [15:52:18] that repo is watching https://github.com/wmde/WikibaseDataModelJavaScript.git [15:52:50] I've asked Phab to update it now [15:53:00] yep, I have my fingers crossed that it's on a 6 hour polling window and we've almost reached it [15:53:02] great! [15:53:03] Last updated Thu, Jun 21, 03:09 (5 h, 43 m ago). [15:53:17] yeah, it was on a longer polling frequency [15:55:01] Thanks both! [16:00:21] tarrow: Last updated Thu, Jun 21, 18:00 (7 s ago). [16:00:31] please check if it's okay now [16:02:00] Hauskatze: It seems to be mirrored on my local copy. I noticed the problem because I was trying to refer to it as a submodule in a jenkins job. Hopefully if it appears here it will work there. Just waiting for jenkins to run. [16:04:05] I guess I couldn't have done the recheck myself without having edit on the phab repo? [16:04:42] (03PS4) 10WMDE-leszek: Remove redundant WikibaseLexeme jobs [integration/config] - 10https://gerrit.wikimedia.org/r/441178 [16:05:37] (03Abandoned) 10WMDE-leszek: Remove redundant WikibaseLexeme jobs [integration/config] - 10https://gerrit.wikimedia.org/r/441181 (owner: 10WMDE-leszek) [16:05:51] (03Abandoned) 10WMDE-leszek: Remove no longer used node selenium + composer job [integration/config] - 10https://gerrit.wikimedia.org/r/441182 (owner: 10WMDE-leszek) [16:09:38] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:11:42] Hi - I just tried to check in the following change: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/TinyMCE/+/441408/ [16:12:12] And it led to this set of errors, which seem to be mostly CSS-related: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/5614/console [16:12:58] These error messages don't seem to relate to the TinyMCE code at all - is this just some temporary glitch with Jenkins and/or core MediaWiki? [16:23:28] (03CR) 10Zfilipin: [C: 031] Remove redundant WikibaseLexeme jobs [integration/config] - 10https://gerrit.wikimedia.org/r/441178 (owner: 10WMDE-leszek) [16:32:33] (03CR) 10Zfilipin: [C: 04-1] Added WikibaseLexeme project for a daily node selenium test run against beta cluster (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/434025 (https://phabricator.wikimedia.org/T194252) (owner: 10WMDE-leszek) [16:36:21] Hauskatze: Yep! All is perfectly working now. Thanks for your help [16:36:30] np :) [16:36:34] glad to help [16:36:40] :) [16:38:43] duh zuul is busy [16:44:50] robh: thanks for adding me to the ldap group. I'm not able to access https://gerrit.wikimedia.org/r/projects/mediawiki/extensions/Flow,dashboards/default which other colleagues can, does it take some time for the changes to propagate? [16:45:15] kostajh hi, you have to be added to the gerrit group for that repo. [16:45:29] indeed, wmf ldap doesnt control those gerrit repos [16:45:40] https://gerrit.wikimedia.org/r/admin/groups/517,members [16:45:56] got it, thanks! [16:46:18] someone should change the owner from admin to project members for https://gerrit.wikimedia.org/r/admin/groups/517 [16:49:05] I tried to check in the TinyMCE extension patch again - same set of errors: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/5618/console [16:49:26] Can anyone help with this? None of these errors look like they come from the TinyMCE code. [17:14:30] Yaron: the last two runs of that job that fail were from that patch: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/ others are passing [17:14:36] Yaron: from the failing test: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/5618/artifact/log/should-have-its-title.png [17:14:47] looks like it's more than just css failures [17:14:58] PROBLEM - Host deployment-deploy-01 is DOWN: CRITICAL - Host Unreachable (10.68.22.177) [17:26:50] greg-g: oh, I missed that, among all the rest. Thanks. [17:28:21] Yaron you have a large topic! [17:28:33] PROBLEM - Puppet errors on deployment-deploy01 is CRITICAL: CRITICAL: 83.33% of data above the critical threshold [0.0] [17:29:17] uh [17:29:30] why is an instance from outside deployment-prep trying to talk to our puppetmaster [17:29:36] krenair@deployment-puppetmaster03:~$ sudo puppet cert list [17:29:36] "fakelogstash.search.eqiad.wmflabs" (SHA256) 95:0D:F5:F3:86:BC:E1:AE:26:95:B4:47:4F:0A:76:38:5D:A3:4C:9C:2E:FF:AD:A3:EF:E4:0B:21:09:FB:9D:5F [17:30:01] -rw-r--r-- 1 puppet puppet 1.6K Jun 14 18:59 /var/lib/puppet/server/ssl/ca/requests/fakelogstash.search.eqiad.wmflabs.pem [17:30:33] paladox: what do you mean? [17:30:48] Yaron "Replaced LinkEnd with HtmlPageLinkRendererEnd hook, for MW > 1.27" [17:30:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561#4305956 (10Krenair) Replacing it myself [17:31:00] that causes a too many query error [17:32:15] It's not that long... it will show up as a complete line in Phabricator. [17:32:36] Wait - this commit message will cause an error, or were you talking about something else? [17:35:00] Yaron it won't cause any noticable errors, but you will get a Server error prompt [17:35:06] i guess this https://gerrit-review.googlesource.com/c/gerrit/+/185690 is what fixes it [17:36:48] paladox: oh... I don't think I've heard of any of these things before - topic queries? Server errors? [17:37:00] Do people query on commit messages? [17:37:15] Yaron https://gerrit.wikimedia.org/r/changes/?O=10b&q=status%3Aopen%20topic%3AReplaced%20LinkEnd%20with%20HtmlPageLinkRendererEnd%20hook%2C%20for%20MW%20%3E%201.27 [17:37:39] "too many terms in query" [17:38:08] Oh... who is querying on commit messages? [17:39:12] the commit msg is not query it i doin't think, but i think it must be the thing that links patches (ie if it has the same topic) [17:39:36] Oh... I get it. [17:40:01] So what's the maximum number of characters/words that should be used? [17:40:25] hmm not sure [17:40:40] Alright. [17:40:45] i guess if you use - [17:40:48] instead of a space [17:40:53] that will work [17:40:57] as that would be one query [17:41:44] Oh... you're talking about the commit ID, not the message. [17:41:55] Yaron nope [17:41:58] Yaron the topic [17:42:10] Sorry, I meant, the branch name. [17:42:16] branch name? [17:42:26] Okay, never mind, you're talking about the commit message. [17:42:44] Yaron https://phabricator.wikimedia.org/F22461576 [17:43:02] But if I put in dashes instead of spaces, it would make it less readable - and queries wouldn't return anything useful anyway. [17:43:33] RECOVERY - Puppet errors on deployment-deploy01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:43:41] gerrit's backend dosen't seem to support lots of topic querys [17:43:49] guess that's why they will strip whitespaces now [17:44:14] Yaron you could use hashtags! [17:45:27] it's not the same thing (you can search by hashtags still) but it won't link patches like topic, but hashtags you can have mutiple ones. [17:45:34] where as topic is limited to one [17:46:39] How would you "hashtag" that commit message I had? [17:46:58] hashtag commit message? [17:47:08] topic and hashtag have nothing to do with the commit message :) [17:47:18] you click add hashtag and enter your term [17:47:45] in gwtui it's "Hashtags" [17:48:43] Alright. [17:48:56] I'll check that out. [18:14:34] PROBLEM - Puppet errors on deployment-deploy01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [18:48:36] thcipriani, around? [18:48:47] Krenair: yeah, what's up? [18:48:56] just re-doing deployment-deploy01 [18:49:06] what was the trick to get past the scap lock due to inactive server problem? [18:50:46] trick was a awful hack in keeping with tradition [18:51:15] I edited GLOBAL_LOCK_FILE in /usr/lib/python2.7/dist-packages/scap/lock.py [18:51:15] well that much I remember :) [18:51:34] and then reran puppet, then changed it back [18:51:44] edited it to what? [18:51:50] just a different path [18:51:55] ok [18:52:01] guess that's logical [18:52:41] it's...uh...pragmatic I guess :\ [18:53:16] have you seen this one? [18:53:21] Notice: /Stage[main]/Profile::Mediawiki::Deployment::Server/Scap::Source[scholarships/scholarships]/Scap_source[scholarships/scholarships]/ensure: created [18:53:21] Error: Execution of '/usr/bin/scap deploy --init' returned 70: 18:53:02 Unhandled error: [18:53:29] Traceback (most recent call last): [18:53:29] File "/usr/lib/python2.7/dist-packages/scap/cli.py", line 336, in run [18:53:29] exit_status = app.main(app.extra_arguments) [18:53:29] File "/usr/lib/python2.7/dist-packages/scap/deploy.py", line 613, in main [18:53:29] self.repo = self.config['git_repo'] [18:53:30] KeyError: 'git_repo' [18:53:31] 18:53:02 deploy failed: 'git_repo' [18:53:56] Error: /Stage[main]/Profile::Mediawiki::Deployment::Server/Scap::Source[dumps/dumps]/Scap_source[dumps/dumps]/ensure: change from absent to present failed: Execution of '/usr/bin/scap deploy --init' returned 70 [18:54:12] I figure that's a problem with dumps/dumps.git config? [18:54:41] there was a problem with the dumps/dumps one where it didn't have the scap repo defined for beta [18:54:49] ah [18:55:31] https://github.com/wikimedia/puppet/blob/production/hieradata/role/common/deployment_server.yaml#L106 [18:55:45] the scholarships one..hrm [18:56:01] I don't think the error was about scholarships thcipriani [18:56:04] I think that was just the line above [18:56:20] puppet very helpfully gave error output and then afterwards told us what the failed resource was [18:56:23] ah, cool :) [18:56:30] because puppet [18:56:37] :D [18:56:51] strange one I noticed scroll by earlier, though fixed now: [18:57:46] https://phabricator.wikimedia.org/P7280 [18:59:08] heh, we should maybe catch that and give a more helpful message [18:59:35] I'm wondering if /etc/dsh/group simply hasn't been populated at that point in time [19:00:17] that's possible, checking too early/ordering problem in puppet sounds like a thing that would fix itself [19:01:00] only after those errors did I later see stuff like: [19:01:01] Notice: /Stage[main]/Scap::Dsh/Scap::Dsh::Group[librenms]/File[/etc/dsh/group/librenms]/ensure: defined content as '{md5}f2465fd75677ab2cc4e5fdf539f7e3fc' [19:01:12] I'm not sure what the relationships are between scap::sources and scap::dsh..yeah that...are [19:01:28] so missing dependencies between stuff in puppet I guess [19:02:00] so right now the errors are just npm package being missing and the git_repo thing for dumps/dumps [19:02:21] going to keep going and add role::aptly::server [19:07:56] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561#4306122 (10Krenair) Alright, should be back to roughly where we were 2 weeks ago now. npm package is still failing,... [19:13:51] I think that means all running beta instances are free of puppet errors [19:14:10] oh no wait because deploy01 still has them [19:14:13] dammit [19:43:25] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:43:50] ugh cache text was failing again [19:44:04] wonder if it's that new acme_tiny [19:45:39] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306202 (10hashar) [19:46:40] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2981228 (10hashar) The builds have completed, I have parsed the results and update the table in this task.... [19:51:12] (03PS1) 10Hashar: Migrate 3 BlueSpice extensions to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441432 (https://phabricator.wikimedia.org/T130811) [19:51:24] PROBLEM - Host deployment-mx is DOWN: CRITICAL - Host Unreachable (10.68.17.78) [19:51:26] (03CR) 10Hashar: [C: 032] Migrate 3 BlueSpice extensions to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441432 (https://phabricator.wikimedia.org/T130811) (owner: 10Hashar) [19:52:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306218 (10hashar) [19:53:03] (03Merged) 10jenkins-bot: Migrate 3 BlueSpice extensions to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441432 (https://phabricator.wikimedia.org/T130811) (owner: 10Hashar) [19:54:24] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:06:20] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-1.diskspace.root.byte_percentfree (<33.33%) [20:07:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306244 (10hashar) [20:25:55] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306308 (10hashar) Digging in the log for the `master`, 34 extensions are failling due to PHP Deprecated er... [20:29:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306321 (10hashar) [20:31:12] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306341 (10hashar) [20:36:55] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306379 (10hashar) [20:49:43] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-2.diskspace.root.byte_percentfree (<20.00%) [21:31:55] root@deployment-deploy01:/srv/deployment/dumps/dumps# grep scap .gitignore [21:31:57] /scap [21:32:02] thcipriani is that supposed to be there? [21:32:14] yeah [21:32:21] oh I see what this is [21:32:28] you wrote "dumps/dumps had no scap directory. Had to clone git clone https://gerrit.wikimedia.org/r/p/operations/dumps/scap.git manually" [21:32:31] but in my case [21:32:34] they have a separate repository for the scap directory that is git ignored [21:32:38] dumps/dumps *has* a scap directory, *but* [21:32:51] I think it must be missing from the beta heiradata [21:32:51] only because something needed to create scap/log/ [21:33:01] right, scap deploy --init did that probably [21:33:14] so if I rm that and get it put in properly, that should solve this problem [21:33:24] yeah, it should [21:33:25] let's see if I can avoid manual git clones [21:34:48] hieradata/role/common/deployment_server.yaml contains this under scap::sources: [21:34:52] dumps/dumps: [21:34:53] repository: operations/dumps [21:34:53] scap_repository: operations/dumps/scap [21:35:10] but hieradata/labs/deployment-prep/common.yaml just has this: [21:35:13] dumps/dumps: [21:35:13] repository: operations/dumps [21:36:39] yeah, I was just making a patch for this :) [21:37:02] https://gerrit.wikimedia.org/r/c/operations/puppet/+/441491 [21:37:48] I think this was in horizon hieradata the first time I setup deployment-deploy01 [21:39:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306499 (10hashar) [21:44:29] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561#4306521 (10Krenair) @thcipriani's https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441491/ will fix broken stu... [21:44:40] PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka-jumbo-2.diskspace.root.byte_percentfree (<30.00%) [21:45:06] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [21:46:25] thcipriani, looks like that did it [21:46:30] nice :) [21:48:19] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306539 (10hashar) [21:49:25] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3067735 (10hashar) I have dropped 34 extensions that do not have a REL1_27 branch (found via T197903). If... [21:52:46] (03PS1) 10Hashar: Migrate BlueSpiceConfigManager to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441493 (https://phabricator.wikimedia.org/T130811) [21:52:48] (03PS1) 10Hashar: Migrate BlueSpiceNamespaceCSS to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441494 (https://phabricator.wikimedia.org/T130811) [21:54:04] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (shipyard), 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#4306566 (10hashar) [21:54:13] (03CR) 10Hashar: [C: 032] Migrate BlueSpiceConfigManager to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441493 (https://phabricator.wikimedia.org/T130811) (owner: 10Hashar) [21:54:19] (03CR) 10Hashar: [C: 032] Migrate BlueSpiceNamespaceCSS to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441494 (https://phabricator.wikimedia.org/T130811) (owner: 10Hashar) [21:56:20] (03Merged) 10jenkins-bot: Migrate BlueSpiceConfigManager to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441493 (https://phabricator.wikimedia.org/T130811) (owner: 10Hashar) [21:56:25] (03Merged) 10jenkins-bot: Migrate BlueSpiceNamespaceCSS to Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/441494 (https://phabricator.wikimedia.org/T130811) (owner: 10Hashar) [22:00:04] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [23:15:35] 10Phabricator: Phabricator says I subscribed myself to a task twice in a row - https://phabricator.wikimedia.org/T197913#4306704 (10matmarex)