[00:03:14] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2450925 (10AlexMonk-WMF) While dealing with the commit above for T129586 I noticed we probably need to rescue these from NFS: ```$wgCaptchaDirectory = '/data/p... [00:27:37] (03CR) 10EBernhardson: Single Line comments no multiple '*'. (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/295895 (owner: 10Lethexie) [00:46:12] Yippee, build fixed! [00:46:13] Project UploadWizard-api-commons.wikimedia.beta.wmflabs.org build #4265: 09FIXED in 11 sec: https://integration.wikimedia.org/ci/job/UploadWizard-api-commons.wikimedia.beta.wmflabs.org/4265/ [02:00:50] 05Continuous-Integration-Scaling, 10Horizon, 06Labs: Labs project admin can not delete per project image on Horizon - https://phabricator.wikimedia.org/T110936#1590423 (10AlexMonk-WMF) ```modules/openstack/files/kilo/glance/policy.json: "delete_image": "rule:admin_or_glanceadmin", modules/openstack/files/... [03:05:42] 06Release-Engineering-Team (Deployment-Blockers), 05Release, 05WMF-deploy-2016-07-05_(1.28.0-wmf.9): MW-1.28.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T138555#2451143 (10Krenair) [04:05:37] Yippee, build fixed! [04:05:37] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #71: 09FIXED in 9 min 36 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/71/ [04:14:27] Krenair: is https://phabricator.wikimedia.org/T119736 really a blocker though? that's been on-going for weeks :/ [04:14:40] scroll down [04:15:04] it skyrocketed the day wmf.9 hit wikipedias [04:15:10] I see the other channel now, too [04:15:45] it's been a problem for 3 months [04:16:10] but it became 10x more frequent with wmf.9 [04:16:38] actually no, problem since november [04:16:53] 3 months is just the log cut off [04:16:58] but yeah, 7/7 a big jump, order of magnitude [04:17:16] UBN for 7 months [04:45:08] 06Release-Engineering-Team (Deployment-Blockers), 05Release, 05WMF-deploy-2016-07-05_(1.28.0-wmf.9): MW-1.28.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T138555#2451228 (10Krenair) 05Resolved>03Open We decided to revert back to wmf.8 due to T119736 - https://gerrit.wikimedia.org/r/298414 [04:47:02] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T139211#2422929 (10Krenair) Schedule now in doubt due to our reverting back to wmf.8 (due to T119736 - https://gerrit.wikimedia.org/r/298414) [06:21:07] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:15:17] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2451540 (10Tobi_WMDE_SW) @zeljkofilipin yes, please delete those tests. I can spend some time to work on fixing Wikidata tests next w... [07:53:46] 06Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10VisualEditor: i18n string not updated in the Hebrew Wikipedia, even though it was translated - https://phabricator.wikimedia.org/T134072#2451654 (10Aklapper) 05Open>03declined Declining as per last comment. :( [08:11:40] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2451706 (10Aklapper) Could someone update the SQL results in P3326? Looks like quite some got fixed in the meantime...... [08:27:21] 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering: Turn on phpcs for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2451786 (10Addshore) [08:30:57] (03PS1) 10Addshore: Turn on phplint for analytics/wmde/scripts [integration/config] - 10https://gerrit.wikimedia.org/r/298423 (https://phabricator.wikimedia.org/T140067) [08:31:22] 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering, 13Patch-For-Review, 03TCB-Team-Sprint-2016-06-29: Turn on phpcs for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2451786 (10Addshore) [08:31:43] 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering, 13Patch-For-Review, 03TCB-Team-Sprint-2016-06-29: Turn on phpcs for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2451786 (10Addshore) a:03Addshore [08:56:14] addshore: around? Do you really want to lint analytics/wmde/scripts with Zend PHP 5.3 ? [08:56:40] addshore: also the task refers to PHPCodeSniffer, would be better done using composer :D [08:57:17] 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering, 13Patch-For-Review, 03TCB-Team-Sprint-2016-06-29: Turn on phplint for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2451887 (10Addshore) [08:57:25] oh wow, yeh, should be 5.5..... [08:58:01] (03PS2) 10Addshore: Turn on phplint for analytics/wmde/scripts [integration/config] - 10https://gerrit.wikimedia.org/r/298423 (https://phabricator.wikimedia.org/T140067) [08:58:05] fixed the task name too! [09:03:03] addshore: for PHPCodeSniffer you will want to turn to composer later on :) [09:03:13] (03CR) 10Hashar: [C: 032] Turn on phplint for analytics/wmde/scripts [integration/config] - 10https://gerrit.wikimedia.org/r/298423 (https://phabricator.wikimedia.org/T140067) (owner: 10Addshore) [09:03:50] addshore: deployed! [09:04:08] (03Merged) 10jenkins-bot: Turn on phplint for analytics/wmde/scripts [integration/config] - 10https://gerrit.wikimedia.org/r/298423 (https://phabricator.wikimedia.org/T140067) (owner: 10Addshore) [09:05:29] hashar ssh is still working with zuul and gerrit :) [09:05:48] https://gerrit-zuul.wmflabs.org/ [09:09:33] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2451937 (10Paladox) @Aklapper I belive I completed all on what's on that list :) But @Dzahn could you check again to m... [09:18:14] 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2451956 (10Paladox) @bd808 we could stop those commits from being linked in tasks through an update to the code but not sure how we do that. @mmodell hi, do you have an idea how we can block refs... [09:38:53] thanks hashar ! [09:48:01] 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - https://phabricator.wikimedia.org/T126552#2452003 (10hashar) The puppet tidy type got merged ( https://gerrit.wikimedia.org/r/#/c/295641 ). I can... [09:48:10] 10Continuous-Integration-Infrastructure, 07Jenkins: Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - https://phabricator.wikimedia.org/T126552#2452004 (10hashar) [09:50:16] 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering, 03TCB-Team-Sprint-2016-06-29: Turn on phplint for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2452007 (10hashar) @addshore for PHPCodeSniffer you will want to use composer and one of the various Jenkins composer test j... [10:11:56] !log Github created repos operations-debs-contenttranslation-apertium-mk-en and operations-docker-images-toollabs-images for Gerrit replication [10:12:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:37:38] PROBLEM - Parsoid on deployment-parsoid06 is CRITICAL: Connection refused [11:31:54] (03PS2) 10Hashar: Migrate coverage reports to doc.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/298276 (https://phabricator.wikimedia.org/T139620) [11:32:08] (03CR) 10Hashar: [C: 032] "Jobs refreshed" [integration/config] - 10https://gerrit.wikimedia.org/r/298276 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [11:33:08] (03CR) 10Hashar: [C: 032] Migrate /cover/ under doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/298275 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [11:33:48] (03Merged) 10jenkins-bot: Migrate coverage reports to doc.wikimedia.org [integration/config] - 10https://gerrit.wikimedia.org/r/298276 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [11:34:06] (03Merged) 10jenkins-bot: Migrate /cover/ under doc.wikimedia.org [integration/docroot] - 10https://gerrit.wikimedia.org/r/298275 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [11:41:23] hashar: it is still doing noops :/ https://gerrit.wikimedia.org/r/#/c/298447/1 [11:41:42] (03PS1) 10Zfilipin: Deleted browsertests-Wikidata* Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/298448 (https://phabricator.wikimedia.org/T128190) [11:41:50] (03PS1) 10Hashar: .htaccess to redirect /cover/ to doc.wm.o [integration/docroot] - 10https://gerrit.wikimedia.org/r/298449 (https://phabricator.wikimedia.org/T139620) [11:42:05] (03CR) 10Hashar: [C: 032] .htaccess to redirect /cover/ to doc.wm.o [integration/docroot] - 10https://gerrit.wikimedia.org/r/298449 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [11:42:15] addshore: oh [11:42:24] (03PS2) 10Zfilipin: Deleted browsertests-Wikidata* Jenkins jobs [integration/config] - 10https://gerrit.wikimedia.org/r/298448 (https://phabricator.wikimedia.org/T128190) [11:42:42] (03Merged) 10jenkins-bot: .htaccess to redirect /cover/ to doc.wm.o [integration/docroot] - 10https://gerrit.wikimedia.org/r/298449 (https://phabricator.wikimedia.org/T139620) (owner: 10Hashar) [11:44:49] addshore: the result does not make sense. gotta investigate :( [11:44:56] okay :( [11:45:07] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-RelatedArticles, 13Patch-For-Review: browsertests-RelatedArticles-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce job fails with: unknown environment `beta` (MediawikiSelenium::ConfigurationError) - https://phabricator.wikimedia.org/T128084#2452231 (10... [11:45:09] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Migration of browsertests* Jenkins jobs to selenium* jobs - https://phabricator.wikimedia.org/T128190#2452230 (10zeljkofilipin) 05Open>03Resolved [11:45:12] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 13Patch-For-Review: browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce fails with "757: unexpected token" - https://phabricator.wikimedia.org/T128079#2452232 (10zeljkofilipin) [11:45:40] addshore: OHH [11:45:47] addshore: and I thought I got the patch deployed [11:45:48] but I hadn't [11:45:52] :D [11:46:10] at least that was an easy fix up! sorry about that [11:46:11] hashar probaly because it is in check-voter [11:46:11] well, at least it isn't some mysterious thing you have to investigate! ;) [11:46:12] https://phabricator.wikimedia.org/rCICF8250cd3621891286e48e3169322e2d353d0794fa [11:46:23] addshore ^^ [11:46:33] paladox: I hadn't deployed the fix [11:46:38] Oh [11:46:45] Basic test build succeeded. [11:46:45] php55lint SUCCESS in 2s [11:47:12] woo! [11:47:14] addshore: and definitely switch to composer, will let you get phpcss as well [11:47:14] :D [11:52:10] 10Continuous-Integration-Config, 15User-zeljkofilipin: Refactor Jenkins job configuration for UploadWizard-api-commons.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T140076#2452246 (10zeljkofilipin) [12:05:39] 10Browser-Tests-Infrastructure, 10MediaWiki-General-or-Unknown, 15User-zeljkofilipin: Run browser tests in core on every commit - https://phabricator.wikimedia.org/T111467#2452309 (10zeljkofilipin) [12:22:33] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #79: 04FAILURE in 31 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/79/ [12:37:19] for god sake [12:37:26] folks updating tasks that are two years old [12:37:56] hashar which ones [12:38:07] random ones [12:38:24] Oh [12:38:38] I think that may be Danny_B [12:39:44] among others yeah [12:39:47] Oh [12:40:26] hashar My whole notification feed is covered with old tasks being updated [12:40:31] In phabricator [12:41:58] hashar im wondering could you build the zuul for jessie the one we found stalled on ssh please [12:42:16] I want to make sure it works correctly and that the test i did is accurate [12:58:30] 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering, 03TCB-Team-Sprint-2016-06-29, 15User-Addshore: Turn on phplint for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2452478 (10Addshore) [13:05:01] paladox: I am busy this week preparing the phase out of gallium / writing doc etc [13:05:08] Oh :) [13:05:15] paladox: maybe I will try to upgrade Zuul again to debug the failure later in the week [13:05:23] Ok [13:06:24] hashar i could see if i could setup another instance [13:06:27] running precise [13:06:34] and gerrit 2.8 and jenkins 1.* [13:06:39] with that zuul version [13:09:08] hashar are you doing the gallium migration this week? [13:24:50] 10Continuous-Integration-Config, 06WMDE-Analytics-Engineering, 03TCB-Team-Sprint-2016-06-29, 15User-Addshore: Turn on phplint for analytics/wmde/scripts repo - https://phabricator.wikimedia.org/T140067#2452627 (10Addshore) p:05Triage>03Normal [13:40:20] hashar mutante proposed https://phabricator.wikimedia.org/T133150#2450319 for doc.wm.org [13:40:58] 07Browser-Tests, 10MobileFrontend, 06Reading-Web-Backlog: `Generic special page features.Search from Watchlist` test failing - https://phabricator.wikimedia.org/T130971#2452776 (10zeljkofilipin) a:03zeljkofilipin [13:41:23] 07Browser-Tests, 10Browser-Tests-Infrastructure, 10MobileFrontend, 06Reading-Web-Backlog, 15User-zeljkofilipin: `Generic special page features.Search from Watchlist` test failing - https://phabricator.wikimedia.org/T130971#2152450 (10zeljkofilipin) [13:42:34] 07Browser-Tests, 10Browser-Tests-Infrastructure, 10MobileFrontend, 06Reading-Web-Backlog, 15User-zeljkofilipin: `Generic special page features.Search from Watchlist` test failing with Net::ReadTimeout - https://phabricator.wikimedia.org/T130971#2152450 (10zeljkofilipin) [13:52:09] 06Release-Engineering-Team, 06Collaboration-Team-Interested, 06Developer-Relations, 06Editing-Department, and 13 others: Create team projects for all teams participating in scrum of scrums - https://phabricator.wikimedia.org/T1211#2452816 (10Danny_B) [13:55:46] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-zeljkofilipin: selenium-Wikibase Jenkins job fails with `no such file to load -- features/support/pages (LoadError)` - https://phabricator.wikimedia.org/T140096#2452882 (10zeljkofilipin) [13:56:21] 10Continuous-Integration-Infrastructure, 06Project-Admins, 10Zuul, 07Jenkins, 15User-greg: Create tag for zuul-cloner - https://phabricator.wikimedia.org/T84945#2452898 (10Danny_B) [13:58:54] 07Browser-Tests, 10Wikidata, 15User-zeljkofilipin: selenium-Wikidata Jenkins job fails with MediawikiSelenium::ConfigurationError - https://phabricator.wikimedia.org/T140097#2452906 (10zeljkofilipin) [14:02:30] 07Browser-Tests, 10Wikidata, 15User-zeljkofilipin: selenium-Wikidata fails with Timeout::Error - https://phabricator.wikimedia.org/T140098#2452925 (10zeljkofilipin) [14:03:10] 07Browser-Tests, 10Browser-Tests-Infrastructure, 10MobileFrontend, 06Reading-Web-Backlog, 15User-zeljkofilipin: `Generic special page features.Search from Watchlist` test failing with Net::ReadTimeout - https://phabricator.wikimedia.org/T130971#2452939 (10zeljkofilipin) p:05Low>03Normal [14:03:28] 07Browser-Tests, 10Wikidata, 15User-zeljkofilipin: selenium-Wikidata fails with Timeout::Error - https://phabricator.wikimedia.org/T140098#2452925 (10zeljkofilipin) p:05Triage>03Normal [14:03:31] 07Browser-Tests, 10Wikidata, 15User-zeljkofilipin: selenium-Wikidata Jenkins job fails with MediawikiSelenium::ConfigurationError - https://phabricator.wikimedia.org/T140097#2452942 (10zeljkofilipin) p:05Triage>03Normal [14:03:34] 10Browser-Tests-Infrastructure, 10Wikidata, 15User-zeljkofilipin: selenium-Wikibase Jenkins job fails with `no such file to load -- features/support/pages (LoadError)` - https://phabricator.wikimedia.org/T140096#2452943 (10zeljkofilipin) p:05Triage>03Normal [14:07:40] 06Release-Engineering-Team, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2452958 (10zeljkofilipin) [14:07:52] 06Release-Engineering-Team, 15User-zeljkofilipin: Update new SWAT member process and deploying documentation in prep for adding European SWAT window - https://phabricator.wikimedia.org/T139545#2452960 (10zeljkofilipin) [14:08:07] 06Release-Engineering-Team, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2452962 (10zeljkofilipin) [14:16:22] scap hasn't broken on beta has it? [14:16:28] Can't see a bot notification [14:17:04] Reedy: https://integration.wikimedia.org/ci/view/Beta/ would know :) [14:17:22] not failed recently [14:17:25] so I didn't break anything [14:17:28] ta [14:18:02] thcipriani: good morning [14:18:12] hashar: howdy [14:18:17] thcipriani: I got the coverage reports moved from integration.wm.o to doc.wm.o with proper 301 redirect :))) [14:18:29] https://integration.wikimedia.org/cover/utfnormal/ [14:18:35] magically redirects to https://doc.wikimedia.org/cover/utfnormal/ [14:18:39] \o/ [14:18:54] awesome! [14:19:07] if you get some minutes ahead, I could use some review / pair chat about moving the doc hosting out of gallium [14:19:10] https://phabricator.wikimedia.org/T137890 [14:19:15] now, as you said in some ticket, we "just" need to move the doc domain :) [14:19:30] I did a veryyyy lame graph overview on the task detail ( gdoc is https://docs.google.com/drawings/d/16aYqCa9nSY6SR1zQazQZ6gVSme-VRm8ObGgPqbjXgeE/edit ) [14:19:34] yeah "just" :] [14:19:42] good to see they are now decoupled [14:20:27] the coupling of zuul/jenkins proxy and generated stuff really worried me after our discussion with chase [14:20:38] I am happy we found an easy fix [14:21:13] so, wrt scandium, chasemp had a discussion (and I think he commented on some ticket) about trying to use contint1001 since we have that machine already. scandium is, evidently, 5 years old. [14:21:45] yeah :( [14:22:06] would just need to change the vlan for contint1001 rather than add new disks on old hardware. [14:22:11] https://phabricator.wikimedia.org/T139938#2447334 [14:22:29] so we can keep scandium as is [14:22:34] rebuild all the stuff of contint1001 [14:22:46] and once happy with it turn the switch [14:23:21] hashar mutante asked https://phabricator.wikimedia.org/T133150#2450319 [14:23:31] for doc.wm.org [14:24:23] paladox: and ? [14:24:28] paladox: have you seen my reply? [14:24:30] :D [14:24:55] thcipriani: then contint1001 is ridiculously overpowered so I thought scandium would be good enough for us [14:25:09] Oh, yep, that the doc.wikimedia.org home is tracked [14:26:07] thcipriani: maybe we could set all that crap in Ganeti VM if they had access to labs support host / labs instances [14:26:46] that would be ideal [14:36:29] Hello [14:36:46] Have you documented the rollback of wmf.8 somewhere? [15:07:02] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2453296 (10fgiunchedi) >>! In T64835#2450341, @AlexMonk-WMF wrote: > @fgiunchedi: Okay, so I think I need your help to finish this off completely. > * Where ar... [15:29:00] I tried to ssh into deployment-bastion earlier :/ [15:36:40] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2453444 (10AlexMonk-WMF) >>! In T64835#2453296, @fgiunchedi wrote: >>>! In T64835#2450341, @AlexMonk-WMF wrote: >> @fgiunchedi: Okay, so I think I need your he... [15:40:26] bd808 we can trun off Publish/Notify in the operations/puppet repo. [15:46:05] Yippee, build fixed! [15:46:06] Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #77: 09FIXED in 24 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/77/ [16:41:06] ostriches hi, with https://phabricator.wikimedia.org/T116246 are you going to disable the web editor or leave it enabled it by default [16:41:15] leaving it enabled will be super useful. [16:41:22] please [16:42:01] I haven't even thought about it. No reason to disable it afaik. [16:42:20] (web-based editor isn't even remotely on my radar, way more important things) [16:43:41] !log deploying f472f65 to ores-beta [16:43:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:47:34] ostriches oh, thanks. It would allow even more people from mobiles to contribute with ease. [16:48:11] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2454046 (10Dzahn) ``` +----------------+-----------------------------------------------------+ | page_namespace | page_... [16:48:12] Gerrit's already pretty craptastic from a phone anyway :p [16:49:05] paladox: just for your information: today is a very busy day for us with things in production (partial outages, lots of users unable to login, etc). So we (releng) are focused on that, not low priority anything else. [16:49:37] greg-g oh sorry, i was just asking. [16:49:46] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2454066 (10Dzahn) note: some things are maybe "won't fix", clickable links should be fixed, but we don't need/should ed... [16:51:34] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2454091 (10Paladox) @dzahn thanks. [16:52:38] I'm having to bring deployment-upload back up to rescue a few files that didn't get migrated properly [16:53:05] * greg-g nods [16:53:07] (It's the only deployment-prep machine left with NFS set up) [16:55:24] RECOVERY - Host deployment-upload is UP: PING OK - Packet loss = 0%, RTA = 1.54 ms [17:16:13] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2454238 (10Aklapper) I edited some. Others are ignorable / intentional. What's left: > | 0 | Deployments... [17:19:48] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2454241 (10Paladox) This task should have it limits to the develepers sites such as mw.org and wikitech and maybe meta.... [17:35:16] 06Release-Engineering-Team, 15User-zeljkofilipin: Update new SWAT member process and deploying documentation in prep for adding European SWAT window - https://phabricator.wikimedia.org/T139545#2454303 (10thcipriani) After a pairing session for SWAT deploy with @zeljkofilipin today, I added some much more detai... [17:39:33] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2454314 (10Dzahn) Yes, i agree wikitech is done and also that the task should be limited to certain wikis we control.... [17:40:34] thcipriani: that /deployers page is great, +1 [17:41:05] greg-g: hopefully it proves more helpful than overwhelming :) [17:41:59] I like the style, it's not overwhelming to me, but maybe I'm not the right test subject :) [17:43:22] E_TOOMANYWORDS [17:43:23] :p [17:43:28] (kidding, it's useful) [17:46:38] heh, the goal would be to get this documentation up the level of https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys which is verbose but really really solid. [17:47:27] and if it's too long then https://wikitech.wikimedia.org/wiki/SWAT_deploys#Doing_the_deploy might be the documentation for you :P [17:57:50] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Setup a Swift cluster on beta-cluster to match production - https://phabricator.wikimedia.org/T64835#2454394 (10AlexMonk-WMF) >>! In T64835#2450925, @AlexMonk-WMF wrote: > Along with this image that probably broke due to my messing around with the config while... [17:58:27] thcipriani: yeah, that's what I was thinking. We should link to it from [[train deploys]] and maybe even [[how to deploy]] [18:01:01] Project mediawiki-core-code-coverage build #2131: 04FAILURE in 3 hr 1 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2131/ [18:35:09] hey wow, that IRC notification actually worked [18:38:16] (03CR) 10Jforrester: "Ping on this (or, at least, the task; it's unclear to me how/whether this achieves that)." [integration/config] - 10https://gerrit.wikimedia.org/r/288320 (https://phabricator.wikimedia.org/T134946) (owner: 10Paladox) [18:38:32] 10Continuous-Integration-Config, 10OOjs-UI, 13Patch-For-Review: Move OOUI out of the MediaWiki gate-and-submit queue - https://phabricator.wikimedia.org/T134946#2454594 (10Jdforrester-WMF) p:05Triage>03Normal [18:59:20] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review, 07Puppet: mediawiki jobs fail intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2454701 (10hashar) a:03JanZerebecki Fixed month ago by @JanZerebecki and... [18:59:24] 10Continuous-Integration-Infrastructure, 07Blocked-on-Operations, 13Patch-For-Review, 07Puppet: mediawiki jobs fail intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" - https://phabricator.wikimedia.org/T126699#2454703 (10hashar) 05Open>03Resolved [18:59:26] 10Continuous-Integration-Infrastructure, 07Developer-notice: Switch MySQL storage to tmpfs - https://phabricator.wikimedia.org/T96230#2454704 (10hashar) [19:59:14] * hasharAway refrains from swearing [19:59:48] * legoktm hugs hasharAway [20:00:10] the autoupgrade of packages on CI permanent slave has been broken for a while :/ [20:04:40] !log Maybe fix unattended upgrade on the CI slaves via https://gerrit.wikimedia.org/r/298568 [20:04:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:16:11] hashar it seems precise is broken in labs so i carn't test zuul with gerrit 2.8 [20:22:02] !log CI force running puppet on all instances: salt --batch 5 -v '*' puppet.run [20:22:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:22:13] paladox: that would not surprise me [20:22:25] lot of puppet code has been dropped iirc [20:22:26] Oh, but i think they know how to fix it [20:22:37] and the labs image has probably been disabled for a while [20:22:41] just it was using the old ldap [20:22:59] and nope you can still install precise [20:23:14] paladox: I am probably going to upgrade Zuul prod tomorrow morning [20:23:22] with some extra debug [20:23:31] hashar ok thanks :) [20:23:31] when I did it weeks ago it was working [20:23:37] Oh [20:23:39] but eventually stall after a while due to some reason [20:23:43] the question is [20:23:50] hashar it seems to be working with gerrit 2.12 [20:23:56] was it a one time weird effect or is the package really bugged [20:24:01] but i havent tested reliably [20:24:03] yeah that is great! [20:24:15] well at least it is not mis behaving ;] [20:24:15] Since you create the dpkg [20:24:19] nope [20:24:40] I did the packaging [20:24:41] on [20:24:44] jenkins-slave-01 [20:25:00] /etc/zuul/zuul1/ [20:25:14] thats ^^ the jessie repo for zuu; [20:25:16] zuul [20:25:23] /etc/zuul/zuul2/ [20:25:38] that's the precise repo ^^ i doint know why i git cloned it [20:25:45] since i just git merged from upstream [20:26:43] paladox: can you possibly document the versions of Gerrit and Zuul you are using on the tes tproject? [20:26:44] on https://phabricator.wikimedia.org/T137525 [20:26:48] hashar im not sure what time your doing it at, but i will see if i will be online early to help [20:26:50] Yep [20:26:50] ok [20:26:52] the package version of zuul [20:26:55] + Gerrit version [20:26:57] Ok [20:26:59] for history purpose :] [20:27:02] Ok [20:27:06] danke!! [20:27:37] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2455183 (10Paladox) Using gerrit 2.12 and zuul version 2.1.0-151-g30a433b-wmf1jessie1 works no ssh problems. [20:27:44] Your welcome and hashar ^^ done [20:28:27] hashar plus that was my first time packaging a dpkg. Theres a wpkg [20:28:28] 2.1.0-151-g30a433b-wmf1jessie1 [20:28:34] Woops wrong text [20:28:38] https://en.wikipedia.org/wiki/Wpkg [20:29:37] !log integration: force running unattended upgrade on all instances: salt --batch 4 -v '*' cmd.run 'unattended-upgrade' . That upgrades diamond and hhvm among others. imagemagick-common has a prompt though [20:29:39] I want to see if i can build a dpkg on windows. [20:29:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:29:47] paladox: wpkg ? [20:29:55] It is what dpkg is [20:29:57] paladox: is that to build a debian package on Microsoft Windows? [20:30:00] Yes [20:30:04] holy shit [20:30:05] and to include a .exe file [20:30:17] so it works on both debian and windows [20:30:26] I am in awe [20:30:34] Oh [20:30:36] Yep [20:30:44] 20 years ago I would have never believed that to become a possibility [20:30:45] microsoft is also making skype for linux [20:30:48] tomarror [20:31:03] yep but now microsoft is open sourcing everything [20:31:11] * hashar applies at Microsoft [20:31:14] Oh [20:31:15] ha [20:31:26] Oo, [20:31:31] But i can run ubuntu unity nativly on windows [20:31:34] without a vm [20:31:45] thx for the comment on the task! [20:31:45] there is limitations of course but it is just wow [20:31:49] your welcome [20:31:49] ""Using gerrit 2.12 and zuul version 2.1.0-151-g30a433b-wmf1jessie1 works no ssh problems."" [20:31:54] Yep [20:31:55] :) [20:32:13] if you get time you could write a quick blog post about how you have managed to rebuild the whole infra on labs [20:32:16] that might be a good show case [20:32:22] Oh [20:32:23] yep [20:32:24] if you have a tech blog [20:32:26] i started that [20:32:31] but not a blog [20:32:35] i wrote a wikitech page [20:32:39] but havent finished yet [20:32:41] https://www.google.co.uk/#q=ubuntu+unity+in+windows&tbm=nws [20:33:09] and maybe we could work with WMF folks in San Francisco to have that published on https://blog.wikimedia.org/c/technology/ :] [20:33:15] https://wikitech.wikimedia.org/wiki/How_to_install_zuul,_gerrit_and_jenkins_on_instances [20:33:22] Oh :) [20:33:29] I can imagine a headline such as "volunteer rebuild a critical piece of the infrastructure in a week sprint" [20:33:49] Oh yep with lots of help from hashar :) [20:34:10] but for one thing i carn't fix is gerrit [20:34:14] "Thanks to our fully OpenSource infrastructure, volunteer Paladox managed to rebuild the whole continuous integration infrastructure in a week sprint. He relied on wmflabs Cloud provided by Wikimedia to volunteer ".... [20:34:15] when i restart it, it fails [20:34:17] something like that :) [20:34:23] yep :) [20:34:26] thanks [20:34:51] ah the wikitech page is neat [20:34:59] are you subscribed to the QA mailling list ? [20:35:07] Not sure i doint think so [20:35:12] i am with wikitect-I [20:35:22] it is a low traffic list talking about QA / CI etc https://lists.wikimedia.org/mailman/listinfo/qa [20:35:31] Oh [20:35:43] Ill subscribe now [20:35:43] yeah wikitech-l is the heavy traffic list for any tech topics [20:35:44] :) [20:35:51] and we have a lot more on narrow topics [20:36:08] eg https://lists.wikimedia.org/mailman/listinfo/labs-announce [20:36:18] Oh [20:36:31] https://lists.wikimedia.org/mailman/listinfo has the full list [20:36:43] Your subscription request has been received, and will soon be acted upon. Depending on the configuration of this mailing list, your subscription request may have to be first confirmed by you via email, or approved by the list moderator. If confirmation is required, you will soon get a confirmation email which contains further instructions. [20:36:44] :) [20:37:10] hashar you can use gpkg keys in gerrit [20:37:22] and you can sign off comments i think too :) [20:38:10] hashar i got it falling onto jenkins-slave-ci. [20:38:37] But i called the user account jenkins instead of jenkins-bot [20:38:57] so im not sure if i could create a patch in integration/config adding repos of the test repo. [20:40:48] yeah the jenkins user names are a bit of a mess [20:41:01] jenkins-bot is to interact with gerrit [20:41:06] jenkins-slave is for the slaves in production [20:41:11] jenkins is for the master on production [20:41:17] might have been added to labs as well [20:41:23] and jenkins-deploy is for the slaves in labs [20:41:24] :( [20:41:36] Yep [20:41:38] Oh [20:41:40] Oh [20:41:42] Yep jenkins is the user in labs [20:41:44] yep [20:41:46] Im going to give wpkg a try [20:41:50] hoping it works [20:41:52] :) [20:41:56] Oh look it downloaded. [20:42:02] * paladox installing wpkg :) [20:42:32] apt-get install wpkg ? [20:42:47] Nope i install through .exe [20:42:57] Actually [20:42:59] wow [20:43:04] it installed a deb package [20:43:40] Output folder: c:\WPKG [20:43:40] Extract: wpkg_static.exe... 100% [20:43:40] Extract: admindb_init.txt... 100% [20:43:40] Created uninstaller: c:\WPKG\uninstall.exe [20:43:40] Create folder: c:\WPKG\packages [20:43:40] Output folder: c:\WPKG\packages [20:43:42] Extract: index.tar.gz [20:43:44] Extract: msvc-runtime_12.0.0_mswindows-amd64.deb [20:43:46] Extract: wpkg-dev_1.0.0_mswindows-amd64.deb... 100% [20:43:48] Extract: wpkg-doc_1.0.0_all.deb... 100% [20:43:50] Extract: wpkg-gui_1.0.0-0_mswindows-amd64.deb... 100% [20:43:52] Extract: wpkg-hdrs-dev_1.0.0_all.deb... 100% [20:43:54] Extract: wpkg_1.0.0_mswindows-amd64.deb... 100% [20:43:56] Extract: wpkggui-doc_1.0.0-0_all.deb... 100% [20:43:58] Extract: wpkgguiqt5_1.0.0-0_mswindows-amd64.deb... 100% [20:44:00] Output folder: c:\WPKG\packages [20:44:04] Output folder: c:\WPKG [20:44:06] Create folder: C:\Users\patrick\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\M2OSW\Windows Packager [20:44:09] Create shortcut: C:\Users\patrick\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\M2OSW\Windows Packager\GUI Windows Packager.lnk [20:44:12] Create shortcut: C:\Users\patrick\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\M2OSW\Windows Packager\Uninstall.lnk [20:44:15] Output folder: c:\WPKG [20:44:17] Output folder: c:\WPKG [20:44:19] Executing command: c:\WPKG\wpkg_static.exe --verbose --root c:\WPKG --create-admindir c:\WPKG\admindb_init.txt [20:44:22] Executing command: c:\WPKG\wpkg_static.exe --verbose --root c:\WPKG --add-sources "wpkg file:///WPKG/packages/ ." [20:44:25] Executing command: c:\WPKG\wpkg_static.exe --verbose --root c:\WPKG --update [20:44:27] wpkg_static.exe:info: 2016/07/12 21:42:59: successfully updated index file from repository: "/WPKG/packages/./index.tar.gz". (repository) [20:44:30] Executing command: c:\WPKG\wpkg_static.exe --verbose --root c:\WPKG --install wpkg wpkg-gui [20:44:35] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate directories (validate-installation) [20:44:36] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate package name (validate-installation) [20:44:38] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate installation type (source/binary) (validate-installation) [20:44:41] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate installed packages (validate-installation) [20:44:43] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate architecture (validate-installation) [20:44:45] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate pre-dependencies (validate-installation) [20:44:47] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate dependencies (validate-installation) [20:44:49] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate distribution name (validate-installation) [20:44:51] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate packager version (validate-installation) [20:44:53] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate fields (validate-installation) [20:44:55] wpkg_static.exe:debug: 2016/07/12 21:42:59: validate size and overwrites (validate-installation) [20:44:57] Woops sorry [20:44:59] I didnt mean to do that [20:45:03] I copied a small bit it seems to have copy the whole thing [20:45:34] hashar ^^7 [20:51:12] * paladox adding C:\WPKG\bin to my path [20:52:10] luckily irssi (my client) warns you, before pasting more than 5 lines :D [20:52:42] Oh [20:52:56] "but for one thing i carn't fix is gerrit / when i restart it, it fails" [20:53:03] Yep [20:53:04] paladox: Check the error log? [20:53:08] I have [20:53:12] nothing showed in there [20:53:25] That's weird. What happens when you try `./bin/gerrit.sh run`? [20:53:33] That should spam it all to the CLI instead of log [20:53:38] Oh let me try that [20:54:21] ./bin/gerrit.sh run [20:54:26] root@gerrit-test:/var/www/html/review_site# ./bin/gerrit.sh run [20:54:26] Running Gerrit Code Review: [20:54:34] [2016-07-12 20:54:21,178] [main] INFO com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache /var/www/html/review_site/cache [20:55:06] maybe it could be because of a warnning about multiple guice things open [20:55:56] Nah it should proceed past that. [20:56:02] Is that all it said? [20:56:29] Yep [20:56:38] Currently it is stuck at that [20:56:45] but then it takes a while then it works [20:56:56] [2016-07-12 20:56:03,707] [main] INFO com.google.gerrit.server.config.ScheduleConfig : changeCleanup schedule parameter "changeCleanup.startTime" is not configured [21:01:37] [2016-07-12 20:59:24,351] [main] INFO com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /var/www/html/review_site/plugins [21:01:37] Jul 12, 2016 8:59:24 PM com.google.inject.servlet.GuiceFilter setPipeline [21:01:37] WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely ignore this message. If this is NOT deliberate however, your application may not work as expected. [21:08:34] paladox: https://phabricator.wikimedia.org/paste/ is your friend ! [21:08:46] Yep [21:08:52] I didnt mean to paste all that [21:10:29] Guice / Gerrit I have no clue Chad is your only hope! [21:10:29] hashar what time are you doing the zuul upgrade tomarror? [21:10:33] Yep [21:10:33] not sure [21:10:38] and ok [21:10:42] 10am CEST probably [21:10:44] aka when I show up [21:10:48] Ok :) [21:10:53] 9 am my time [21:11:01] hashar wow wpkg is a dpkg [21:11:04] will give it a try for an half hour and revert back to known version [21:11:13] going to head to bed [21:11:20] have good hacking [21:11:23] Ok bye [21:11:31] theres spam in -operations [21:16:12] ostriches i got it working http://gerrit-test.wmflabs.org/gerrit/ [21:16:18] normaly i have to do it twice [21:16:28] then wait for it to show as working at http://gerrit-test.wmflabs.org/gerrit/ [21:16:44] Yeah it's probably slow to startup on labs. [21:16:54] If you're using puppet though, those new config things should be set and not warn [21:17:05] Oh, i set those configs [21:17:08] role::gerrit::server *SHOULD* work in labs now [21:17:11] in the gerrit.config [21:17:14] oh [21:17:23] I rewrote it to use SSL with letsencrypt :) [21:17:30] Instead of trying to not. [21:17:30] Oh :) [21:17:50] I got zuul working too [21:18:24] and jenkins so upgrading shoulden affect those but my testing is not that reliable since im using newer software. [21:20:16] ostriches im currently creating https://wikitech.wikimedia.org/wiki/How_to_install_zuul,_gerrit_and_jenkins_on_instances (Havent finished though) [21:21:03] I know. [21:21:14] That's why I mentioned the gerrit manifest. [21:21:18] I wasn't seeing how it got installed [21:22:02] 10Continuous-Integration-Infrastructure, 10Zuul: Investigate Zuul 2.1.0-151-g30a433b that stops processing Gerrit events - https://phabricator.wikimedia.org/T137525#2455399 (10Paladox) p:05Low>03Normal Changing to normal so we can do some debugging which we are doing tomarror. Let's hope it was a one time... [21:22:50] Oh [21:23:13] I installed it all manually except from using some of puppet rules for gerrit proxy and zuul [21:32:30] 10Deployment-Systems, 06Release-Engineering-Team, 07WorkType-NewFunctionality: Use subrepos instead of git submodules for deployed MediaWiki extensions - https://phabricator.wikimedia.org/T98834#2455478 (10Jdforrester-WMF) [21:58:07] Project selenium-Core » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #84: 04FAILURE in 6 min 6 sec: https://integration.wikimedia.org/ci/job/selenium-Core/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/84/ [22:11:23] 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, and 2 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2455647 (10thcipriani) >>! In T129151#2441318, @elukey wrote: > The analytics refinery (https://phabricator.wikimedia.org/diffusion/ANRE/browse/mast... [22:27:21] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2455734 (10Danny_B) Maybe also foundation wiki in case there are some links there... As it is "official wiki". [22:28:37] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2455750 (10Dzahn) doesn't look like there is anything https://wikimediafoundation.org/w/index.php?search=git&title=Sp... [22:31:39] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2455767 (10Dzahn) meta, 27 hits: https://meta.wikimedia.org/w/index.php?title=Special:Search&search=git.wikimedia.org&... [22:44:50] 05Gitblit-Deprecate, 10Diffusion, 13Patch-For-Review: Update all references to git.wikimedia.org and replace them with the Phabricator equivalent - https://phabricator.wikimedia.org/T137353#2455824 (10Danny_B) I've got much more on Meta actually: 7 @ https://meta.wikimedia.org/w/index.php?target=git.wikimedi... [23:45:32] ostriches when you have time could you change the default branch at https://github.com/wikimedia/phabricator to wmf/stable please [23:47:49] Did the same for /arcanist too [23:48:19] And libphutil [23:48:33] arcanist was probably a good one, in case people clone from us there [23:49:10] Oh [23:49:20] Thanks [23:49:21] :)