[00:20:07] PROBLEM - Puppet run on deployment-phab02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:22:55] PROBLEM - Puppet run on deployment-phab01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:27:52] (03PS1) 10Tim Starling: Add ParserMigration [tools/release] - 10https://gerrit.wikimedia.org/r/344280 [00:33:49] Yippee, build fixed! [00:33:49] Project mediawiki-core-code-coverage build #2651: 09FIXED in 3 hr 4 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2651/ [00:56:24] (03CR) 10Reedy: [C: 032] Add ParserMigration [tools/release] - 10https://gerrit.wikimedia.org/r/344280 (owner: 10Tim Starling) [00:57:10] (03Merged) 10jenkins-bot: Add ParserMigration [tools/release] - 10https://gerrit.wikimedia.org/r/344280 (owner: 10Tim Starling) [00:59:31] Reedy: I still have to add the submodules to the existing deployment branches manually, right? [00:59:51] TimStarling: If you're wanting it in .16 or .17 yeah [01:00:09] If you're gonna wait for .18, you can just let it appear magically from next week [01:01:36] Presume you're not gonna bother with .16 at least as that goes away tomorr [01:02:37] ok, so .18 would mean full deployment a week from tomorrow [01:03:17] yup, indeed [01:08:39] apparently I neglected to update composer.json when I fixed a bug in the library [01:09:06] so probably best to wait for .18, otherwise I would need to run composer update in .17 [01:11:41] composer update for what? [01:12:40] RemexHtml, actually I didn't even push the tag [01:13:14] this extension, ParserMigration, provides an edit/preview page which compares two different Tidy implementations [01:13:27] Guess it needs to be higher than 1.0.0 https://github.com/wikimedia/mediawiki-vendor/blob/master/composer.json#L73 [01:13:42] the two are the existing tidy, and RemexHtml [01:13:54] yeah, 1.0.1 doesn't even exist yet in composer [01:13:58] in packagist I mean [01:14:19] FWIW, if anyone else is likely to use the extension... ParserMigration/composer.json should really have a require section that has RemexHtml in it too [01:14:42] Then the merge plugin can bring it in if anyone is using that and also the extension [01:17:00] the library is in core because I think it's suitable as a default tidier [01:17:19] oh, duh [01:17:24] I didn't even look there :) [01:18:04] the next step for it might be to use it in the initial LocalSettings.php written by the installer [01:18:41] the first step for WMF is content migration, fixing any broken HTML that relies on tidy in odd ways [01:18:56] then the next step will be to use it for default page views [01:19:12] Have you got any automated ways for finding the broken HTML? [01:19:20] I suspect there's gonna be quite a few bits [01:20:48] if commonly used templates are broken then they will show up in visual diff testing, which uses a random selection of articles [01:21:20] for things like missing end tags, we have https://www.mediawiki.org/wiki/Extension:Linter which was recently deployed [01:22:20] visual diff testing indicates that it is a small proportion of pages with breaking changes [01:22:27] still probably thousands of pages, but it is manageable [01:24:07] Generally, people seem very good at learning to fish [01:24:21] It'll just be smaller wikis that probably need people to explicitly go out and sort them [01:27:28] 06Release-Engineering-Team, 07Puppet: Preload TestingAccessWrapper in production mwrepl - https://phabricator.wikimedia.org/T143607#3124033 (10Mattflaschen-WMF) [01:38:47] 10Gerrit, 06Operations, 10Ops-Access-Requests: Add two Analytics team members to wmf-deployments - https://phabricator.wikimedia.org/T161157#3124038 (10Dzahn) [01:43:49] (03PS1) 10EddieGP: Remove support for old branch names [tools/release] - 10https://gerrit.wikimedia.org/r/344286 [01:44:19] 10Gerrit, 06Operations, 10Ops-Access-Requests: Add two Analytics team members to wmf-deployments - https://phabricator.wikimedia.org/T161157#3124056 (10Dzahn) done in gerrit web ui. (per: milimetric has existing shell with deployment access, ottomata has root) [01:47:21] 10Gerrit, 06Operations, 10Ops-Access-Requests: Add two Analytics team members to wmf-deployments - https://phabricator.wikimedia.org/T161157#3124076 (10Dzahn) 05Open>03Resolved a:03Dzahn [02:35:30] (03CR) 10Reedy: [C: 032] Remove support for old branch names [tools/release] - 10https://gerrit.wikimedia.org/r/344286 (owner: 10EddieGP) [02:37:51] (03Merged) 10jenkins-bot: Remove support for old branch names [tools/release] - 10https://gerrit.wikimedia.org/r/344286 (owner: 10EddieGP) [04:17:53] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #339: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/339/ [06:34:10] Yippee, build fixed! [06:34:10] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #308: 09FIXED in 1 hr 54 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/308/ [06:50:57] Yippee, build fixed! [06:50:57] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #308: 09FIXED in 2 hr 10 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/308/ [09:14:37] Project beta-code-update-eqiad build #148285: 15ABORTED in 1 min 36 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/148285/ [09:17:35] PROBLEM - jenkins_zmq_publisher on contint1001 is CRITICAL: connect to address 127.0.0.1 and port 8888: Connection refused [09:19:35] RECOVERY - jenkins_zmq_publisher on contint1001 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 8888 [09:32:41] ^^^ Jenkins restart [10:14:27] (03PS1) 10Jonas Kress (WMDE): Enable experimental browsertests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/344339 (https://phabricator.wikimedia.org/T161201) [10:23:29] (03CR) 10Hashar: [C: 032] Enable experimental browsertests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/344339 (https://phabricator.wikimedia.org/T161201) (owner: 10Jonas Kress (WMDE)) [10:25:01] (03Merged) 10jenkins-bot: Enable experimental browsertests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/344339 (https://phabricator.wikimedia.org/T161201) (owner: 10Jonas Kress (WMDE)) [10:27:23] (03CR) 10Hashar: "Deployed :}" [integration/config] - 10https://gerrit.wikimedia.org/r/344339 (https://phabricator.wikimedia.org/T161201) (owner: 10Jonas Kress (WMDE)) [10:29:55] (03PS3) 10Hashar: Add non-voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/344162 (owner: 10Umherirrender) [10:33:57] (03CR) 10Hashar: [C: 032] Add non-voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/344162 (owner: 10Umherirrender) [10:34:18] (03PS4) 10Hashar: Add non-voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/344162 (owner: 10Umherirrender) [10:34:30] (03CR) 10Hashar: [C: 032] Add non-voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/344162 (owner: 10Umherirrender) [10:35:44] (03Merged) 10jenkins-bot: Add non-voting unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/344162 (owner: 10Umherirrender) [10:58:29] (03CR) 10Hashar: "Looks like the situation is slightly better since yesterday. Note that the ongoing issue impacts (impacted?) pretty much every jobs :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/343738 (owner: 10Ladsgroup) [11:08:58] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: Create "High Priority" test pipeline - https://phabricator.wikimedia.org/T160667#3124562 (10hashar) On the [[ https://grafana-admin.wikimedia.org/dashboard/db/zuul | Grafana Zuul board ]] I have added a graph showing the time to... [11:09:14] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: Create "High Priority" test pipeline - https://phabricator.wikimedia.org/T160667#3124563 (10hashar) [11:11:27] 10Continuous-Integration-Config: Remove check-voter pipeline from Zuul - https://phabricator.wikimedia.org/T161205#3124578 (10hashar) [11:12:36] (03PS1) 10Hashar: [operations/dns] switch to test-prio pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/344351 (https://phabricator.wikimedia.org/T160667) [11:16:58] (03CR) 10Hashar: [C: 032] [operations/dns] switch to test-prio pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/344351 (https://phabricator.wikimedia.org/T160667) (owner: 10Hashar) [11:17:58] (03Merged) 10jenkins-bot: [operations/dns] switch to test-prio pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/344351 (https://phabricator.wikimedia.org/T160667) (owner: 10Hashar) [11:20:38] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Create "High Priority" test pipeline - https://phabricator.wikimedia.org/T160667#3124610 (10hashar) [11:21:19] 10Gerrit, 10Analytics-Tech-community-metrics: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3124611 (10Aklapper) [11:21:34] 10Gerrit, 10Analytics-Tech-community-metrics: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3124611 (10Aklapper) [11:21:38] 10Gerrit, 10Analytics-Tech-community-metrics: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3124625 (10Aklapper) [11:21:50] 10Gerrit, 10Analytics-Tech-community-metrics: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3124625 (10Aklapper) [11:22:12] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Create "High Priority" test pipeline - https://phabricator.wikimedia.org/T160667#3107070 (10hashar) Status ==== Three repositories now benefit from the high priority pipeline: * operations/mediawiki-config * operations/puppet * operations/dns Todo... [11:27:14] (03PS1) 10Jonas Kress (WMDE): Enable QUnit tests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/344356 [11:27:29] (03PS2) 10Jonas Kress (WMDE): Enable QUnit tests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/344356 [12:32:41] (03CR) 10Hashar: [C: 032] Enable QUnit tests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/344356 (owner: 10Jonas Kress (WMDE)) [12:33:36] (03Merged) 10jenkins-bot: Enable QUnit tests for WikibaseLexeme [integration/config] - 10https://gerrit.wikimedia.org/r/344356 (owner: 10Jonas Kress (WMDE)) [12:38:14] 10Gerrit, 10Analytics-Tech-community-metrics: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3124814 (10hashar) [12:39:26] 10Gerrit, 10Analytics-Tech-community-metrics: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3124815 (10hashar) [12:40:32] 10Gerrit, 10Analytics-Tech-community-metrics: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3124625 (10hashar) I can reach them all now. Seems all those patches are drafts. Most probably tightly related to the ot... [12:44:17] 10Gerrit, 10Analytics-Tech-community-metrics: Gerrit patchset 99101 cannot be accessed: "500 Internal server error" - https://phabricator.wikimedia.org/T161206#3124611 (10hashar) gerrit show-caches has: ``` Name |Entries | AvgGet |Hit Ratio|... [13:29:31] 10Gerrit, 06Operations, 10Ops-Access-Requests: archiva-deploy password for Chad H. - https://phabricator.wikimedia.org/T161067#3120509 (10MoritzMuehlenhoff) Yeah, but that requires the setup of a second pwstore repo, since the current one for ops is on a restricted host. [13:47:04] Yippee, build fixed! [13:47:04] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #345: 09FIXED in 3 min 3 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/345/ [14:02:43] PROBLEM - Host deployment-ms-be01 is DOWN: CRITICAL - Host Unreachable (10.68.16.24) [14:02:57] !log deployment-ms-be01 and deployment-ms-be02 : Lower Swift replicator on, upgrade package, reboot hosts. T160990 [14:03:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:03:01] T160990: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990 [14:06:06] 10Gerrit, 10Analytics-Tech-community-metrics: Numerous Gerrit patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3125089 (10Aklapper) Interesting. I still cannot reach them (logged in on Firefox, logged out on Chromium)... So maybe th... [14:07:46] RECOVERY - Host deployment-ms-be01 is UP: PING OK - Packet loss = 0%, RTA = 2.28 ms [14:10:39] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:12:38] 10Beta-Cluster-Infrastructure, 10media-storage, 13Patch-For-Review: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3125117 (10hashar) Each instance uses 300% user CPU and 100% system CPU. So potentially 8 core... [14:20:43] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:23:29] 10Continuous-Integration-Config, 13Patch-For-Review: Phase out jobs "pplint-HEAD" and "erblint-HEAD" - https://phabricator.wikimedia.org/T154894#3125154 (10hashar) Last patches have been merged by @Ottomata :-} [14:40:39] (03PS1) 10Hashar: zuul: remove check-voter pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/344394 (https://phabricator.wikimedia.org/T161205) [14:41:07] I somehow ran into the archived repo USERINFO in gerrit yesterday. It had Owner:Registered Users set, which means everybody was still able to contribute and code review in it although it's archived. I changed that to Owner:Administrators , I hope that's okay. [14:41:49] eddiegp: eeeek [14:41:56] but it is set read-only in Gerrit! [14:42:15] Yep, but I could have simply changed that back ;) [14:42:25] oh yeah [14:42:57] (03CR) 10Hashar: [C: 032] zuul: remove check-voter pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/344394 (https://phabricator.wikimedia.org/T161205) (owner: 10Hashar) [14:44:33] (03Merged) 10jenkins-bot: zuul: remove check-voter pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/344394 (https://phabricator.wikimedia.org/T161205) (owner: 10Hashar) [14:45:59] 10Continuous-Integration-Config, 13Patch-For-Review: Remove check-voter pipeline from Zuul - https://phabricator.wikimedia.org/T161205#3125260 (10hashar) 05Open>03Resolved a:03hashar [14:48:17] (03PS1) 10Hashar: Move DonationInterface + REL1_28 job to experimental [integration/config] - 10https://gerrit.wikimedia.org/r/344395 (https://phabricator.wikimedia.org/T160476) [14:49:19] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10FR-Smashpig, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, and 2 others: Disable fundraising CI jobs that are non-voting and always fail - https://phabricator.wikimedia.org/T160476#3125276 (10hashar) [14:49:56] (03CR) 10Hashar: [C: 032] Move DonationInterface + REL1_28 job to experimental [integration/config] - 10https://gerrit.wikimedia.org/r/344395 (https://phabricator.wikimedia.org/T160476) (owner: 10Hashar) [14:50:52] (03Merged) 10jenkins-bot: Move DonationInterface + REL1_28 job to experimental [integration/config] - 10https://gerrit.wikimedia.org/r/344395 (https://phabricator.wikimedia.org/T160476) (owner: 10Hashar) [14:54:37] PROBLEM - Free space - all mounts on deployment-ores-redis is CRITICAL: CRITICAL: deployment-prep.deployment-ores-redis.diskspace._srv.byte_percentfree (<100.00%) [14:55:55] 10Gerrit, 06Operations, 10Ops-Access-Requests: Add two Analytics team members to wmf-deployments - https://phabricator.wikimedia.org/T161157#3125292 (10Milimetric) Thank you very much [14:56:57] (03PS1) 10Hashar: [DonationInterface] skip generic job on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/344400 (https://phabricator.wikimedia.org/T160476) [14:57:31] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10FR-Smashpig, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, and 2 others: Disable fundraising CI jobs that are non-voting and always fail - https://phabricator.wikimedia.org/T160476#3125297 (10hashar) [14:58:18] (03CR) 10Hashar: [C: 032] [DonationInterface] skip generic job on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/344400 (https://phabricator.wikimedia.org/T160476) (owner: 10Hashar) [14:59:12] (03Merged) 10jenkins-bot: [DonationInterface] skip generic job on master branch [integration/config] - 10https://gerrit.wikimedia.org/r/344400 (https://phabricator.wikimedia.org/T160476) (owner: 10Hashar) [15:01:04] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10FR-Smashpig, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, and 2 others: Disable fundraising CI jobs that are non-voting and always fail - https://phabricator.wikimedia.org/T160476#3125303 (10hashar) 05Open>03Resolved a... [15:31:15] (03PS5) 10Ejegg: Use upstream civicrm-buildkit [integration/config] - 10https://gerrit.wikimedia.org/r/336960 [15:31:55] (03PS1) 10Hashar: Experimental job that merges puppet.git jobs [integration/config] - 10https://gerrit.wikimedia.org/r/344404 (https://phabricator.wikimedia.org/T160923) [15:32:27] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10OOjs-UI: Speed up oojs/ui Jenkins jobs - https://phabricator.wikimedia.org/T155483#3125384 (10hashar) a:05hashar>03None [15:32:37] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: For operations/puppet : merge tox / rake jobs in a single job? - https://phabricator.wikimedia.org/T160923#3114888 (10hashar) a:03hashar [15:52:38] !log Deleting integration-slave-trusty-1011 m1.large. One less perm slave to take care about [15:52:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:02] PROBLEM - Host integration-slave-trusty-1011 is DOWN: CRITICAL - Host Unreachable (10.68.17.244) [16:04:55] (03CR) 10Hashar: [C: 032] Experimental job that merges puppet.git jobs [integration/config] - 10https://gerrit.wikimedia.org/r/344404 (https://phabricator.wikimedia.org/T160923) (owner: 10Hashar) [16:05:50] 10Gerrit, 06Release-Engineering-Team, 13Patch-For-Review: Update gerrit to 2.14 - https://phabricator.wikimedia.org/T156120#3125450 (10Paladox) They have branched stable-2.14 now. I've back ported my patches for polygerrit to that branch. The private changes will be in gerrit 2.15+ as they have just landed t... [16:06:40] (03Merged) 10jenkins-bot: Experimental job that merges puppet.git jobs [integration/config] - 10https://gerrit.wikimedia.org/r/344404 (https://phabricator.wikimedia.org/T160923) (owner: 10Hashar) [16:06:45] (03PS1) 10Hashar: Merge operations/puppet jobs [integration/config] - 10https://gerrit.wikimedia.org/r/344410 (https://phabricator.wikimedia.org/T160923) [16:07:08] !log restbase deploying 752ca4b7 [16:07:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:07:23] ups wrong chan [16:12:32] (03CR) 10Hashar: [C: 032] Merge operations/puppet jobs [integration/config] - 10https://gerrit.wikimedia.org/r/344410 (https://phabricator.wikimedia.org/T160923) (owner: 10Hashar) [16:13:57] (03Merged) 10jenkins-bot: Merge operations/puppet jobs [integration/config] - 10https://gerrit.wikimedia.org/r/344410 (https://phabricator.wikimedia.org/T160923) (owner: 10Hashar) [16:21:27] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: For operations/puppet : merge tox / rake jobs in a single job? - https://phabricator.wikimedia.org/T160923#3125476 (10hashar) 05Open>03Resolved Tested, works. I have update the castor cache by rebuilding a build with ZUUL_P... [16:27:46] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10media-storage, 13Patch-For-Review: deployment-ms-be01.deployment-prep and deployment-ms-be02.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3125491 (10hashar) [16:38:28] (03PS1) 10Aleksey Bekh-Ivanov (WMDE): Enable experimental browsertests for WikibaseLexeme with composer [integration/config] - 10https://gerrit.wikimedia.org/r/344416 [16:53:36] (03PS1) 10EddieGP: Remove old USERINFO tool [tools/release] - 10https://gerrit.wikimedia.org/r/344418 [17:00:45] thcipriani: will the train get deployed everywhere today? or just group1? [17:03:49] legoktm: I'm going to try to get it out everywhere today. Ran into some problems on Tuesday unfortunately wanted to let it bake on group0 for a while to make sure problems are resolved. [17:04:07] ok, thanks [17:04:28] (03CR) 10Hashar: [C: 032] Enable experimental browsertests for WikibaseLexeme with composer [integration/config] - 10https://gerrit.wikimedia.org/r/344416 (owner: 10Aleksey Bekh-Ivanov (WMDE)) [17:05:23] (03Merged) 10jenkins-bot: Enable experimental browsertests for WikibaseLexeme with composer [integration/config] - 10https://gerrit.wikimedia.org/r/344416 (owner: 10Aleksey Bekh-Ivanov (WMDE)) [17:55:37] (03PS1) 10Hashar: Delete operations-puppet-typos [integration/config] - 10https://gerrit.wikimedia.org/r/344444 (https://phabricator.wikimedia.org/T119140) [18:46:18] https://integration.wikimedia.org/zuul/ [18:46:28] This doesn't look good :D [19:02:16] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T160549#3126015 (10thcipriani) [19:38:10] (03PS1) 10Umherirrender: [MessageCommons] Add npm job [integration/config] - 10https://gerrit.wikimedia.org/r/344476 [20:20:15] twentyafterfour: thx for the badge btw :) [20:20:28] :) [20:22:51] also thanks too :) [20:28:48] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.29.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T160549#3126267 (10jmatazzoni) [20:49:45] (03CR) 10Hashar: [C: 032] Delete operations-puppet-typos [integration/config] - 10https://gerrit.wikimedia.org/r/344444 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [20:50:09] one less job on permanent slaves :) [20:50:41] (03Merged) 10jenkins-bot: Delete operations-puppet-typos [integration/config] - 10https://gerrit.wikimedia.org/r/344444 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [21:08:02] hashar i have a great idea. What about if there is no nodepool instances available when a test run, they could run on permenant instantces as a backup. [21:10:11] 10Gerrit, 06Operations, 10Ops-Access-Requests: archiva-deploy password for Chad H. - https://phabricator.wikimedia.org/T161067#3120509 (10RobH) I think its fine to share, but if its not an emergency is it ok to just get this approved in our ops meeting next Monday? When its approved, then we can just gpg en... [21:17:56] (03CR) 10Hashar: "Fonction is now handled by the already existing operations-puppet-test-jessie job which already runs rake." [integration/config] - 10https://gerrit.wikimedia.org/r/344444 (https://phabricator.wikimedia.org/T119140) (owner: 10Hashar) [21:18:18] paladox: yeah potentially :D [21:18:27] Yep :) [21:18:36] paladox: but the aim was to get rid of the permanent slaves so we no more have to firefight with them [21:19:05] did you get Polygerrit prefix option to be merged upstream ? :} [21:19:19] Nope, i am waiting for them to merge it [21:19:31] hashar they branched stable-2.14 now [21:19:34] but i cherrypicked my patches onto it :) [21:19:41] guess some will have to test your change [21:19:48] ohh [21:19:50] maybe that is going to annoy them ? [21:20:23] Dought it, they haven't told me i am annoying them. [21:20:44] hashar but others have cherry picked things from master too [21:21:04] Anyways, my patch is live on https://gerrit-new.wmflabs.org/r/ :) [21:21:36] hashar i found performance was very good much faster then gwt. 100% improvement on gerrit-new using polygerrit. [21:22:00] I guess the server just send the data [21:22:09] and your local browser ends up doing all the formatting locally [21:22:18] oh [21:22:18] so you save all the round trips [21:22:24] :) [21:22:25] I have no idea how it works really :-} [21:22:31] it uses the rest api [21:22:35] yeah so [21:22:39] that is the browser being the gui [21:22:46] Oh :) [21:23:00] though i found ios 10.3 is broken with polygerrit. [21:23:04] and give the hundred of millions of dollars / countless man hours invested in making browser lightning fast ... [21:23:14] yep [21:23:15] ios 10.2 works with it. [21:23:18] that is surely faster than the decade old / server side GWT [21:23:31] yeh [21:23:31] oh [21:23:32] it deffitly is [21:23:33] + faster diff's too [21:23:37] so it is mobile friendly isn't it ? [21:23:45] It is mobile frendly [21:23:52] Just a bug was introduced in ios 10.3 [21:23:54] breaking it [21:24:23] Reported it here https://bugs.chromium.org/p/gerrit/issues/detail?id=5715 [21:24:35] and https://bugs.webkit.org/show_bug.cgi?id=169970 [21:25:36] hashar it's very mobile friendly too. Also desktop scalable too. [21:26:44] I also found out you can name your patches too. They are calling it descriptions (not to be confussed with commit msg) [21:28:11] * hashar tries [21:29:09] on Android / Firefox that looks like the desktop version [21:29:10] :( [21:29:16] Oh [21:29:46] It is mobile friendly on the iphone [21:32:55] * hashar tries to upload a file to phabricator [21:33:32] :0 [21:33:33] :) [21:33:51] paladox: https://phabricator.wikimedia.org/F6902412 [21:34:01] ah [21:34:02] I see [21:34:15] hashar try https://gerrit-new.wmflabs.org/r/?polygerrit=1 [21:34:24] the photo your showing is your still using gwt. [21:35:16] * hashar feels old [21:35:24] I just filled "https://gerrit-new.wmflabs.org/r/" in the address bar [21:35:51] yep. Polygerrit isen't the default ui yet. Still has alot of work to do before it does. [21:36:08] It needs to implement a project page gui. Admin gui. [21:36:13] Add the inline edit [21:36:58] and the "New UI" link at the bottom [21:37:06] Yep, that's broken. [21:37:07] points e to /q/status:open <-- 404 [21:37:27] but but [21:37:31] yep. I found where it does it in the source code (java) Just doint know how to fix that. [21:37:37] haven't you said that instance has your patch applied ? :D [21:37:45] yep [21:37:50] But it dosen't fix the footer [21:37:59] ah so your Change in upstream need an iteration to fix that footer link right ? [21:38:24] Well kind of yes. I can do that in a seperate change as i haven't figured out how to fix the footer yet. [21:38:38] and once on polygerrit/mobile interface there is an "Old UI" link that is broken as well [21:38:55] points me to /r/#/r/q/status:open/ [21:38:58] yep, but the Old UI button at least correctly takes you back to gwt. [21:40:14] * hashar is happily browsing gerrit on his mobile [21:40:24] that looks like bugzilla, but at least it is readable [21:40:47] :) [21:41:15] Polygerrit is broken on my iphone anyways as it uses ios 10.3 beta. [21:41:29] and I am giving you a token for the links on the Login page [21:41:37] that let us login with a single click [21:41:45] 06Release-Engineering-Team, 15User-greg, 07Wikimedia-Incident: Identify "first responders" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#3126773 (10greg) [21:42:11] :) [21:42:55] hashar you can also set your status in polygerrit [21:42:59] for example away [21:43:02] or busy [21:43:15] you can type anything into that status box [21:43:46] 06Release-Engineering-Team, 15User-greg, 07Wikimedia-Incident: Identify "first responders" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#3126792 (10greg) 05Open>03Resolved Since the last comment on this task there has been a lot of positive edits on mw:D... [21:43:59] adding a review on a change crashed my poor mobile firefox :( [21:44:13] oh. [21:44:24] Were you using gwt or polygerrit? [21:44:28] polygerrit [21:44:36] happened when filling a comment on a change [21:44:44] hmm, does it keep crashing [21:44:48] I sent them the bug report + url [21:44:53] :) [21:45:30] oh [21:45:36] and now listing the open changes [21:46:04] when I click on a change some send me to /q/....something.. URL which is a 404 [21:46:20] Oh. [21:46:31] and it crashed again when trying to reply on a change [21:46:36] but that must be a bug in firefox [21:46:41] Let me do an update on gerrit-new [21:46:59] but overall progress! [21:47:11] * paladox runs git pull && bazel build release [21:47:18] hehe [21:47:23] can't test further though. It is getting late [21:48:36] Ok [21:51:40] 10Deployment-Systems, 06Release-Engineering-Team, 15User-greg: Require an associated task with each SWAT item - https://phabricator.wikimedia.org/T145255#3126857 (10greg) p:05Normal>03Lowest meh? [21:52:12] 06Release-Engineering-Team, 15User-greg: Create FY1617Q1 personal goals (for RelEng team members) - https://phabricator.wikimedia.org/T134518#3126861 (10greg) 05Open>03declined [22:02:04] hashar i've updated gerrit-new now :) [22:06:29] Any Git/Gerrit expert around? "git clone ssh://username@gerrit.wikimedia.org:29418/openzim.git" leads to "warning: remote HEAD refers to nonexistent ref, unable to checkout." Is that fixable server-side? [22:06:39] * andre__ might be in the wrong channel to ask though. Hmm. [22:07:27] andre__: Any way [22:07:28] https://phabricator.wikimedia.org/diffusion/GOZI/ [22:07:30] There's nothing there [22:07:35] here or -operations, but Chad isn't here. He's out today through Tues (back Wed) [22:07:36] Do you want a master creating? [22:07:45] I can reproduce that. [22:07:51] We know there's a problem. [22:08:04] Reedy: Yeah, I got an email by the maintainer that something went wrong ;-( [22:08:11] so I'll file a task, alright [22:08:14] Something went wrong? [22:08:18] We don't populate the repo [22:08:21] I can fix it now [22:08:49] Reedy, if you manage to "restore" I guess that Kelson owes you more than a beer :P [22:08:59] Hang on... [22:09:05] Was there something in the repo previously? [22:09:08] Or was it a new repo? [22:09:32] https://github.com/wikimedia/openzim [22:09:36] there was [22:10:18] To quote from that maintainer's email to me: "Not sure exactly what happened, but I see no way to restore it from my end." [22:10:30] ... [22:10:35] So they did something? [22:10:52] I need to know if they fucked up, or gerrit fucked up [22:11:00] If it's data corruption, I'm not touching it [22:11:10] If they fucked up with git.. I can push the github repo back into it [22:11:37] I wonder would the git log keep this data? [22:11:46] I mean for showing branches being deleted [22:12:02] Reedy: Makes sense. As I'm just a proxy here I'll file a task, CC you and the maintainer, and also reply to their email by pointing to the task. Thanks already so far :) [22:12:10] phabricator shows no branches or tags [22:12:26] I would've presumed it would've replicated down to github though... [22:12:50] https://phabricator.wikimedia.org/diffusion/GOZI/history/project.config/;refs/meta/config [22:12:50] Reedy not really [22:12:54] nothing reent in there [22:12:59] gerrit dosen't force mirror [22:13:05] phabricator on the other hand does [22:13:18] Depends what you mean by force [22:13:26] Whether the action was a force action by the user [22:13:28] https://gerrit.wikimedia.org/r/#/admin/projects/openzim,tags shows the tags [22:13:31] git push --mirror [22:14:22] https://phabricator.wikimedia.org/diffusion/GOZI/ shows no tags at all [22:14:23] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3126908 (10Aklapper) [22:15:07] I've replied to the email and asked the maintainer to clarify in that task. Thanks already :) [22:16:00] Reedy git fsck --lost-found [22:16:08] I'm not running random commands [22:16:16] No not to run on gerrit [22:16:19] rm -rf / [22:16:30] oops, wrong window! ;) [22:16:30] Just saying that brings the branch back for me locally [22:16:44] paladox, feel free to add a comment to that task [22:16:50] ok [22:17:16] alright, sorry for disturbing and thanks everybody for their input. Now back to channel topic. Hopefully. :) [22:17:25] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3126945 (10Reedy) @Kelson did you do something on the repo? Any sort of write/push action? We need to know if it's gerrit related corruption, or if a user did it. If it's the latter, we can push bac... [22:17:36] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3126908 (10Paladox) Hi, running this command "git fsck --lost-found" brought the master branch back for me. $ git fsck --lost-found notice: HEAD points to an unborn branch (master) Checking object d... [22:18:53] $ git fsck --lost-found [22:18:53] notice: HEAD points to an unborn branch (master) [22:18:53] Checking object directories: 100% (256/256), done. [22:18:53] Checking objects: 100% (4067/4067), done. [22:18:55] Does nothing for me [22:19:08] git branch -a returns nothing [22:19:37] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3126950 (10Paladox) strange it shows ^^ but it fails to checkout that branch. [22:19:48] Yep just noticed it my self. [22:20:06] the output of the git clone suggests the objects are still there [22:20:14] so it doesn't feel like data corruption [22:20:20] Someone has just deleted all the branches/tags it seems [22:21:19] Chad is active on phab it seems [22:22:50] fatal: reference is not a tree: f0a79a0f68d376224d72bcca812818c46ca2fb51 [22:24:59] Reedy if this was data corruption it would more likly have prevented us from visiting the project page. [22:28:04] Reedy i can visit refs/meta/config https://phabricator.wikimedia.org/diffusion/GOZI/history/project.config/;refs/meta/config [22:28:20] Reedy: he's just ranting before he's on vacation for a few days :) [22:28:38] paladox: Both are things that I've already said/linked [22:28:38] Reedy https://phabricator.wikimedia.org/rGOZIf0a79a0f68d376224d72bcca812818c46ca2fb51 [22:28:42] ah [22:29:06] greg-g: I thought he was just bored and using airplane wifi ;P [22:29:59] yeah, not sure :) [22:31:14] 10Gerrit: Content of Openzim repository accidentially somehow deleted - https://phabricator.wikimedia.org/T161264#3127007 (10Paladox) I can reach the commit by doing https://phabricator.wikimedia.org/diffusion/GOZI/browse/master/;f0a79a0f68d376224d72bcca812818c46ca2fb51 and https://phabricator.wikimedia.org/rGOZ... [22:31:17] I think he must've arrived... He posted 8 hours. His flight isn't that long [22:41:07] twentyafterfour hi, after git pulling on phabricator again. Im getting [22:41:07] Elasticsearch index Incorrect [22:41:31] I tryed deleting the index and recreating it which does not fix the problem [22:41:44] the status on https://phab-01.wmflabs.org/config/cluster/search/ shows failed [22:41:59] https://phab-01.wmflabs.org/config/issue/elastic.broken-index/ [22:47:38] ElasticSearch http localhost 9200 /phabricator 5 read, write Failed [22:56:58] twentyafterfour maybe it was caused by https://secure.phabricator.com/D17384?vs=42183&id=42191&whitespace=ignore-most#toc ?