[00:47:44] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3590296 (10demon) >>! In T175288#3589585, @RobH wrote: > I'm going to install with jessie, like tin and naos both presently have... [00:57:19] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3590306 (10bd808) Moving the deploy server to stretch should probably wait for a stretch HHVM build shouldn't it? [01:05:32] 10Deployment-Systems, 10Release-Engineering-Team: It shouldn't be possible to create WMF branches on master - https://phabricator.wikimedia.org/T175324#3590314 (10Legoktm) [01:08:44] (03PS1) 10Gergő Tisza: Sniff & fix lowercase @inheritdoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/376654 [02:01:59] (03CR) 10Legoktm: [C: 04-1] "Can you add tests? See MediaWiki/Tests/files/ :)" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/376654 (owner: 10Gergő Tisza) [02:22:59] (03PS1) 10MaxSem: make-wmf-branch: update .gitreview [tools/release] - 10https://gerrit.wikimedia.org/r/376658 (https://phabricator.wikimedia.org/T175324) [02:24:34] PROBLEM - Puppet staleness on deployment-kafka01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [03:32:02] (03CR) 10Chad: "I'm not sure tweaking gitreview fixes the bug here we don't actually use git-review when creating the branch. I purposefully ripped this c" [tools/release] - 10https://gerrit.wikimedia.org/r/376658 (https://phabricator.wikimedia.org/T175324) (owner: 10MaxSem) [03:36:32] 10Gerrit, 10Patch-For-Review, 10User-Ladsgroup: Make gerrit use the new WMF logo - https://phabricator.wikimedia.org/T174576#3590433 (10demon) Considering I don't see the logo that's clearly not going to work. And no, we can't just put our logo alongside any random font--we do have branding guidelines. The l... [03:40:04] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3590434 (10demon) Probably a decent idea. Ignore my idea. [04:58:48] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [04:59:20] (03PS2) 10Gergő Tisza: Sniff & fix lowercase @inheritdoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/376654 [05:18:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [05:52:05] (03CR) 10Legoktm: [C: 032] "Thanks!" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/376654 (owner: 10Gergő Tisza) [05:52:50] (03Merged) 10jenkins-bot: Sniff & fix lowercase @inheritdoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/376654 (owner: 10Gergő Tisza) [08:43:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:04:41] (03PS1) 10Hashar: mwext-testextension-php55-composer to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/376673 (https://phabricator.wikimedia.org/T161882) [09:26:30] (03CR) 10Hashar: [C: 04-1] "Wikibase and any extension depending it on it, end up with:" [integration/config] - 10https://gerrit.wikimedia.org/r/376673 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [09:27:02] 10Release-Engineering-Team, 10ProofreadPage, 10Regression: Proofread buttons stopped working since the deployment of wmf.17 - https://phabricator.wikimedia.org/T175340#3590772 (10Ladsgroup) [09:27:17] halfak: I think this is UBN! https://phabricator.wikimedia.org/T175340 [09:28:23] hashar: I meant you D: [09:40:05] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [integration/jenkins] - 10https://gerrit.wikimedia.org/r/376236 (owner: 10Hashar) [09:40:07] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [tools/release] - 10https://gerrit.wikimedia.org/r/376240 (owner: 10Hashar) [09:44:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Performance-Team, 10Availability (Multiple-active-datacenters): Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#3590845 (10Gilles) [09:44:47] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Performance-Team, 10Availability (Multiple-active-datacenters): Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#2818204 (10Gilles) [11:35:56] (03PS2) 10Hashar: Some mwext-testextension-php55-composer to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/376673 (https://phabricator.wikimedia.org/T161882) [12:17:11] (03CR) 10Hashar: [C: 032] Some mwext-testextension-php55-composer to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/376673 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [12:18:07] (03Merged) 10jenkins-bot: Some mwext-testextension-php55-composer to jessie [integration/config] - 10https://gerrit.wikimedia.org/r/376673 (https://phabricator.wikimedia.org/T161882) (owner: 10Hashar) [12:19:58] AbuseFilter breakage, probably also needs attention / investigation: https://phabricator.wikimedia.org/T175338 [12:20:08] ^ greg-g [12:20:58] ...and https://phabricator.wikimedia.org/T171027 remains a constant source of criticism on Village Pumps. [12:22:50] Yippee, build fixed! [12:22:51] Project selenium-GettingStarted » firefox,beta,Linux,BrowserTests build #519: 09FIXED in 49 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/519/ [12:28:53] Amir1, :P [12:29:01] * halfak wakes up to UBN message :) [12:29:07] :)))) [12:29:08] Sorry [12:29:34] No worries. It happens. :D [12:43:10] andre__: I got a patch for the abuse filter unknown colum task ( https://phabricator.wikimedia.org/T175338 ) [12:43:44] andre__: and for the wathclist, yeah the feature is broken :] [12:52:05] hashar: oh yay thanks. I like patches! :-* [13:46:38] Yippee, build fixed! [13:46:39] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #516: 09FIXED in 2 min 37 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/516/ [14:04:56] 10Release-Engineering-Team (Next), 10Release Pipeline: Proof helm as a potential k8s deployment tool - https://phabricator.wikimedia.org/T173129#3591518 (10Joe) I did a review of how helm works/what it offers in relation to our environment. While I favour using it instead of creating our own tool, there are... [14:05:07] 10Release-Engineering-Team (Next), 10Release Pipeline, 10User-Joe: Proof helm as a potential k8s deployment tool - https://phabricator.wikimedia.org/T173129#3591519 (10Joe) [14:08:26] 10Release-Engineering-Team (Next), 10Release Pipeline, 10User-Joe: Prove helm as a potential k8s deployment tool - https://phabricator.wikimedia.org/T173129#3591587 (10Reedy) [14:27:49] (03CR) 10Hashar: [C: 032] "Puppet patch got merged https://gerrit.wikimedia.org/r/#/c/374999/" [integration/config] - 10https://gerrit.wikimedia.org/r/375001 (owner: 10Hashar) [14:29:44] (03Merged) 10jenkins-bot: dib: mediawiki::packages::php5 is already included [integration/config] - 10https://gerrit.wikimedia.org/r/375001 (owner: 10Hashar) [14:40:39] 10Continuous-Integration-Config, 10MediaWiki-extensions-LabeledSectionTransclusion, 10Patch-For-Review: Load /tests/parser/ParserTestParserHook.php when testing extensions - https://phabricator.wikimedia.org/T166480#3591698 (10hashar) 05Open>03Resolved a:03hashar [14:47:06] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:49:36] (03PS5) 10Umherirrender: Add sniff to find and replace deprecated constants [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375723 (owner: 10Legoktm) [14:49:40] (03CR) 10Umherirrender: [C: 032] Add sniff to find and replace deprecated constants [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375723 (owner: 10Legoktm) [14:50:28] (03Merged) 10jenkins-bot: Add sniff to find and replace deprecated constants [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/375723 (owner: 10Legoktm) [15:05:32] 10MediaWiki-Codesniffer: phpcs fails to put space between require_once and __DIR__ - https://phabricator.wikimedia.org/T175371#3591752 (10Paladox) [15:14:05] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqdfw: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3591796 (10RobH) So Chris flipped around SDA and SDB but the installer still gives an error for SDA not being present. This is likely... [15:14:14] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqdfw: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3591799 (10RobH) [15:16:33] 10MediaWiki-Codesniffer: suggest proper ordering for doxygen tags - https://phabricator.wikimedia.org/T175374#3591805 (10MarkAHershberger) [15:22:08] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:30] (03PS1) 10Hashar: zuul: periodic pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/376740 (https://phabricator.wikimedia.org/T93414) [15:45:09] (03CR) 10Hashar: [C: 04-1] "I have no idea what kind of data the zuul timer send. Does it have ZUUL_BRANCH set? :)" [integration/config] - 10https://gerrit.wikimedia.org/r/376740 (https://phabricator.wikimedia.org/T93414) (owner: 10Hashar) [15:46:31] (03CR) 10jerkins-bot: [V: 04-1] zuul: periodic pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/376740 (https://phabricator.wikimedia.org/T93414) (owner: 10Hashar) [15:55:19] 10Release-Engineering-Team (Watching / External), 10Operations, 10ops-eqdfw: setup/install/deploy deploy1001 as deployment server - https://phabricator.wikimedia.org/T175288#3591915 (10RobH) Chris also reseated it all, still error. [16:07:12] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Developer-Relations (Jul-Sep 2017), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3591940 (10zeljkofilipin) @Rfarrand friendly reminder that we still need a date for this. :) [17:07:04] Hi rel-eng! [17:07:23] Is there some CI-wide composer cache in use? [17:08:13] I'm getting a build failure that seems like composer install is trying to use an older version of the package than exists on packagist [17:08:30] it's a dev-master version requirement, which is the root of the problem [17:08:59] but with a composer clear-cache in the right place, I think it should be able to build [17:09:22] https://integration.wikimedia.org/ci/job/mwext-donationinterfacecore-REL1_27-zend56-jessie/244/console [17:09:30] https://integration.wikimedia.org/ci/job/mwext-donationinterfacecore-REL1_27-zend56-jessie/245/console [17:09:47] happening on two different ci-jessie-wikimedia-XXXXXX instances ^^^^ [17:12:40] ejegg: If you stick in a newer hash, it should help at least confirm that [17:12:53] But AFAIK, yes, there is some composer cache in play (or is it just the web proxy?) [17:13:22] or both even.. [17:13:38] thanks Reedy, I'll make a trivial amendment and see what happens [17:18:15] ejegg: certainly for other stuff.. [17:18:16] 17:13:53 [102.2MB/4.64s] - Installing jakub-onderka/php-parallel-lint (v0.9.2) [17:18:16] 17:13:53 [102.4MB/4.65s] Loading from cache [17:18:44] heh, the patch is still a WIP, I should just be patient and let the cache expire [17:36:54] ejegg: well the cache won't expire for 6 months [17:37:12] whoa! [17:37:27] well, which cache? [17:37:33] the package listing should be ~5 min [17:37:49] but the zip files where it says "Loading from cache" are 6 months I think [17:38:08] huh, I'm pretty sure a 6 month old version of dev-master SmashPig would have broken another build before [17:38:15] link to the patch? [17:39:01] the weird thing is, this patch should have triggered the problem: https://gerrit.wikimedia.org/r/#/c/376717/ [17:39:25] so the SmashPig library used to depend on php-queue: dev-master [17:39:53] there's now a 1.0 version of php-queue, so this morning I updated smash-pig to depend on that version [17:40:05] and updated the dev-master branch on packagist [17:40:25] now I'm updating DonationInterface, which uses smash-pig as a library [17:40:40] DonationInterface still points to smash-pig: dev-master [17:40:54] but it doesn't need to depend directly on php-queue: dev-master any more [17:41:13] hmmm [17:41:28] (03PS1) 10Umherirrender: Add missing unit test and npm jobs [integration/config] - 10https://gerrit.wikimedia.org/r/376761 [17:41:35] composer install is fine locally, and was fine on CI for the patch that actually changed that bit [17:42:28] But the follow-on patch, which swaps out a totally unrelated library, is getting a failure that says smash-pig: dev-master depends on php-queue: dev-master, and violates our minimum-stability [17:43:26] so, maybe it's composer caches on the individual instances [17:45:27] 10MediaWiki-Codesniffer: suggest proper ordering for doxygen tags - https://phabricator.wikimedia.org/T175374#3591805 (10Legoktm) Yep, this should be possible to do in our FunctionCommentSniff rules. [17:52:04] 10Browser-Tests-Infrastructure, 10Release-Engineering-Team (Kanban), 10Developer-Relations (Jul-Sep 2017), 10User-zeljkofilipin: WebdriverIO tech talk - https://phabricator.wikimedia.org/T171852#3592271 (10Rfarrand) Thanks for the reminder, I have sent an email. [17:54:07] ejegg: hmm, maybe something somewhere is setting minimum-stability: stable which prevents usage of dev stuff? [17:54:12] but that should have alerted earlier.. [17:55:22] i think that's the default, right? Then you have to explicitly specify dev-master to accept dev versions [17:56:01] so if I put the php-queue: dev-master dependency back in DonationInterface's composer.json, it would still install with the cached smash-pig [17:56:44] but the whole idea here was to get at least php-queue onto a numbered version [17:57:26] and it's turned into an illustration of why smash-pig oughtta get version numbers too... [19:04:51] Thcipriani, so it looks like you may indeed be able to just add me to the org on docker hub [19:07:57] no_justification: this https://gerrit-review.googlesource.com/#/c/gerrit/+/126072/ will help when it comes to the migration to notedb (even though its a while a way) [19:08:23] since we use puppet, notedb before that change writes to gerrit.config when it finishes, but puppet would overwrite it. [19:08:33] (03CR) 10Umherirrender: Add missing unit test and npm jobs (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/376761 (owner: 10Umherirrender) [19:17:20] addshore: just have to create a docker hub org, currently just a user :\ [19:19:29] Ooooooh [19:19:43] Didn't realise it wasn't an org, I think they look the same from the outside [19:24:15] alright, I managed to convert it to an org https://success.docker.com/Cloud/Solve/How_do_I_convert_my_Docker_Hub%2F%2FCloud_user_account_into_an_organization_account%3F [19:24:33] I'm the owner...so now just have to figure out how to add you :) [19:25:32] addshore: are you addshore on docker hub, too? [19:28:41] Yup [19:29:51] you're added [19:33:31] Awesome, I'll check it out in the next days! [19:41:39] (03PS2) 10Hashar: zuul: periodic pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/376740 (https://phabricator.wikimedia.org/T93414) [19:43:01] (03CR) 10jerkins-bot: [V: 04-1] zuul: periodic pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/376740 (https://phabricator.wikimedia.org/T93414) (owner: 10Hashar) [19:45:33] (03PS3) 10Hashar: zuul: periodic pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/376740 (https://phabricator.wikimedia.org/T93414) [20:07:59] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#3592841 (10Paladox) @demon should we try merging the patch for logstash again? https://gerrit.wikimedia.org/... [20:16:52] thcipriani: how hard would it be to make a memc06,memc07 in deployment prep? [20:20:43] AaronSchulz: hopefully not too terribly difficult (should just be spinning up instances and applying puppet roles). [20:21:12] "should" [20:21:44] "should" and "just" being the operative words [20:22:05] AaronSchulz: what would they be fore? [20:22:22] s/fore/for/ [20:22:34] thcipriani: to replicate from the other two via mcrouter (which would be on all 4) [20:23:07] I want to test around with mcrouter striping/replication [20:23:38] relaying purges across DC's would need this kind of setup [20:23:39] if you file a task for it with the beta-cluster-infrastructure we can try to spin a few up [20:23:51] *beta-cluster-infrastructure tag [20:24:37] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#3592862 (10demon) I guess. Or just abandon the project. I haven't had time to care enough and nobody really l... [20:26:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Availability (Multiple-active-datacenters): Create new instances memc05 and memc06 running memcached - https://phabricator.wikimedia.org/T175418#3592865 (10aaron) [20:26:40] ok [20:31:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Regularly run mwext-{name}-testextension-* jobs to make sure they are still passing after core or dependency changes - https://phabricator.wikimedia.org/T93414#3592882 (10hashar) I am pretty sure @Reedy also as... [21:04:13] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#2494353 (10Dzahn) Checked the firewall setup to make sure that wasn't our blocker (anymore) before we try it... [21:05:28] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#3592970 (10Dzahn) also for gerrit2001, fwiw, we could try from there first, as paladox suggested ``` [gerri... [21:08:35] PROBLEM - Puppet errors on deployment-memc07 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0] [21:08:37] PROBLEM - Puppet errors on deployment-memc07 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [0.0] [21:12:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Wiki-Loves-Monuments-Database, 10User-JeanFred: Automate deployment of heritage on Gerrit post-merge - https://phabricator.wikimedia.org/T157893#3592978 (10greg) [21:19:45] 10Release-Engineering-Team (Kanban), 10User-greg: Distill OCG background into retrospective etherpad - https://phabricator.wikimedia.org/T174735#3592987 (10greg) 05Open>03Resolved my part is done [21:21:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Availability (Multiple-active-datacenters): Create new instances memc05 and memc06 running memcached - https://phabricator.wikimedia.org/T175418#3592865 (10thcipriani) Added 2 new memcached instances: deployment-memc06 and dep... [21:21:31] but the invite still worked [21:21:39] wrong window :) [21:23:21] https://tools.wmflabs.org/versions/ isn't responding... [21:24:22] greg-g: slow but now i got something [21:24:46] 1.30.0-wmf.17 for all groups [21:25:21] mutante: it is fairly slow its usually instant for me [21:26:11] mutante: yeah, odd [21:26:36] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.30.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T170635#3593021 (10greg) 05Open>03Resolved [21:28:34] RECOVERY - Puppet errors on deployment-memc07 is OK: OK: Less than 1.00% above the threshold [0.0] [22:26:58] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Wikimedia-Logstash, 10Patch-For-Review, 10Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#3593150 (10Dzahn) Also tried the log4j.properties config change on gerrit2001 and restarted service.. while i... [23:10:39] 10Beta-Cluster-Infrastructure, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 10MW-1.30-release-notes (WMF-deploy-2017-08-08_(1.30.0-wmf.13)), 10Patch-For-Review: "Loss of session data" on Beta Cluster - https://phabricator.wikimedia.org/T172560#3593235 (10Etonkovidova)... [23:35:24] anyone mind if i deploy a small config change? We put out a survey with sampling rates based on page view counts, but the sampling rates were inverted. e.g. a page that should get 100 impressions and has 5k page views a week got min(1, 5000 / 100) = 1 sampling rate, instead of min(1, 100 / 5000) = .02 [23:35:34] its just changing some numbers, not code or anything