[00:07:08] Any ideas why this patch is stuck and will not merge? https://gerrit.wikimedia.org/r/#/c/224546/ [00:07:23] We're a bit SOL, cos we've locked ourselves out of manual V+2 in this repo [00:08:35] nvm, we worked around it by flushing repeatedly. [00:30:39] (03PS1) 10Aude: Update Wikidata to wmf/1.26wmf16 [tools/release] - 10https://gerrit.wikimedia.org/r/227380 [00:32:02] (03CR) 10Aude: [C: 032] Update Wikidata to wmf/1.26wmf16 [tools/release] - 10https://gerrit.wikimedia.org/r/227380 (owner: 10Aude) [00:38:04] (03Merged) 10jenkins-bot: Update Wikidata to wmf/1.26wmf16 [tools/release] - 10https://gerrit.wikimedia.org/r/227380 (owner: 10Aude) [00:41:58] (03PS1) 10Legoktm: Use npm for UrlShortener [integration/config] - 10https://gerrit.wikimedia.org/r/227383 [00:44:33] (03CR) 10Legoktm: [C: 032] Use npm for UrlShortener [integration/config] - 10https://gerrit.wikimedia.org/r/227383 (owner: 10Legoktm) [00:45:58] (03Merged) 10jenkins-bot: Use npm for UrlShortener [integration/config] - 10https://gerrit.wikimedia.org/r/227383 (owner: 10Legoktm) [00:46:38] !log deploying https://gerrit.wikimedia.org/r/227383 [00:46:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:17:56] PROBLEM - Puppet failure on deployment-restbase01 is CRITICAL 100.00% of data above the critical threshold [0.0] [01:40:14] "At Quora, we generally do post-commit code reviews. That is, the code goes out live in production first and someone comes and reviews the code later." http://engineering.quora.com/Moving-Fast-With-High-Code-Quality [01:55:24] post-commit & deploy privacy review doesn't make me want to use quora much [03:11:27] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1487317 (10BBlack) [03:11:29] 10Beta-Cluster, 10Parsoid, 5Patch-For-Review, 7Varnish: deployment-parsoidcache02 fails varnish VCL compilation - https://phabricator.wikimedia.org/T106662#1487315 (10BBlack) 5Open>3Resolved a:3BBlack [04:44:16] !bash GO DOUSE YOURSELF IN SUGAR AND DIVE INTO AN ANT HILL, JENKINS [04:45:09] !bash GO DOUSE YOURSELF IN SUGAR AND DIVE INTO AN ANT HILL, JENKINS [05:44:31] (03CR) 10Awight: "I had pushed a workaround for the crm job, but can confirm that the new job you deployed still calls "composer install":" [integration/config] - 10https://gerrit.wikimedia.org/r/221310 (owner: 10Awight) [06:38:18] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [06:59:30] RECOVERY - Free space - all mounts on deployment-videoscaler01 is OK All targets OK [07:49:54] 10Browser-Tests, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: investigate failing Wikidata browsertests on jenkins - https://phabricator.wikimedia.org/T92619#1487527 (10Tobi_WMDE_SW) [09:18:32] 6Release-Engineering, 6Phabricator: Evaluate Releeph for SWAT queue management - https://phabricator.wikimedia.org/T101794#1487644 (10Qgil) [09:30:40] 10Beta-Cluster, 10Parsoid, 5Patch-For-Review, 7Varnish: deployment-parsoidcache02 fails varnish VCL compilation - https://phabricator.wikimedia.org/T106662#1487687 (10hashar) @BBlack you are a hero! [09:40:14] !log rebooting deployment-apertium01 to ensure its ferm rules are properly loaded on boot ( https://phabricator.wikimedia.org/T106658 ) [09:40:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:49:56] 10Beta-Cluster: Upgrade beta cluster puppet master from Precise to Trusty - https://phabricator.wikimedia.org/T106649#1487717 (10hashar) [09:49:57] 10Beta-Cluster: Can not ssh to beta cluster instance deployment-apertium01 - https://phabricator.wikimedia.org/T106658#1487714 (10hashar) 5Open>3Resolved a:3hashar **TL;DR: ferm still present with outdated conf files. Removed ferm** Ah I tried accessing with salt yesterday but it did not work for some re... [09:50:33] !log deployment-apertium01 is back! The ferm rules were outdated / not maintained by puppet, dropped ferm entirely. [09:50:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:34:19] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1487796 (10hashar) 5Resolve... [10:42:46] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1487801 (10hashar) The node... [11:04:17] (03PS1) 10Paladox: Update HitCounters tests [integration/config] - 10https://gerrit.wikimedia.org/r/227438 [11:04:29] (03PS2) 10Paladox: Update HitCounters tests [integration/config] - 10https://gerrit.wikimedia.org/r/227438 [11:06:07] (03PS3) 10Paladox: Update HitCounters tests [integration/config] - 10https://gerrit.wikimedia.org/r/227438 [11:12:28] !log Jenkins jobs for the beta cluster ended up stuck again. Found a workaround by removing the Jenkins label on deployment-bastion node and reinstating it. Seems to get rid of the deadlock ( ref: https://phabricator.wikimedia.org/T72597#1487801 ) [11:12:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:14:42] (03PS4) 10Paladox: Update HitCounters tests [integration/config] - 10https://gerrit.wikimedia.org/r/227438 [11:18:34] !log Assigning label "BetaClusterBastion" to https://integration.wikimedia.org/ci/computer/deployment-bastion.eqiad/ [11:18:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:22:56] (03PS1) 10Hashar: beta: expand {datacenter} to 'eqiad' [integration/config] - 10https://gerrit.wikimedia.org/r/227440 (https://phabricator.wikimedia.org/T72597) [11:22:58] (03PS1) 10Hashar: beta: disambiguate Jenkins label from node name [integration/config] - 10https://gerrit.wikimedia.org/r/227441 (https://phabricator.wikimedia.org/T72597) [11:28:38] (03CR) 10Hashar: [C: 032] "noop change since datacenter only had "eqiad"" [integration/config] - 10https://gerrit.wikimedia.org/r/227440 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [11:30:47] (03Merged) 10jenkins-bot: beta: expand {datacenter} to 'eqiad' [integration/config] - 10https://gerrit.wikimedia.org/r/227440 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [11:39:01] (03CR) 10Hashar: [C: 032] "I have added the BetaClusterBastion on https://integration.wikimedia.org/ci/computer/deployment-bastion.eqiad/ and refreshed the jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/227441 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [11:41:27] (03Merged) 10jenkins-bot: beta: disambiguate Jenkins label from node name [integration/config] - 10https://gerrit.wikimedia.org/r/227441 (https://phabricator.wikimedia.org/T72597) (owner: 10Hashar) [11:56:42] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#1487946 (10hashar) [11:57:15] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Jenkins, 7Upstream: [upstream] Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597#747903 (10hashar) I renamed... [12:07:18] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [12:45:30] PROBLEM - Free space - all mounts on deployment-videoscaler01 is CRITICAL deployment-prep.deployment-videoscaler01.diskspace._var.byte_percentfree (<40.00%) [13:18:37] (03CR) 1020after4: [C: 032] Check for l10n cache before sync-wikiversions [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) (owner: 1020after4) [13:19:00] (03Merged) 10jenkins-bot: Check for l10n cache before sync-wikiversions [tools/scap] - 10https://gerrit.wikimedia.org/r/226353 (https://phabricator.wikimedia.org/T100573) (owner: 1020after4) [13:25:16] (03PS2) 10Hashar: Throttle mediawiki core jobs to one per node [integration/config] - 10https://gerrit.wikimedia.org/r/227234 [13:34:15] 10Deployment-Systems, 5Patch-For-Review: Make sync-wikiversions check that a valid localisation cache exists when syncing new versions - https://phabricator.wikimedia.org/T100573#1488040 (10mmodell) 5Open>3Resolved [13:38:53] (03CR) 10Hashar: "That would help reducing the amount of disk space consumed on slaves. Wikidata builds are already throttled." [integration/config] - 10https://gerrit.wikimedia.org/r/227234 (owner: 10Hashar) [13:40:44] (03PS2) 10Hashar: Adjust build/artifact retention for mw-selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/226650 (https://phabricator.wikimedia.org/T104583) (owner: 10Dduvall) [13:42:06] (03CR) 10Hashar: [C: 032] "Thanks to have taken care of it :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/226650 (https://phabricator.wikimedia.org/T104583) (owner: 10Dduvall) [13:45:05] (03Merged) 10jenkins-bot: Adjust build/artifact retention for mw-selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/226650 (https://phabricator.wikimedia.org/T104583) (owner: 10Dduvall) [13:46:16] 10Continuous-Integration-Infrastructure, 7Zuul: Bump python-gear package to 0.5.7 - https://phabricator.wikimedia.org/T98294#1488064 (10hashar) a:5hashar>3None [13:47:39] hashar, do you know what deployment-cache-text03 does? [13:47:57] that is a varnish "text" cache [13:48:07] yes, I know that much [13:48:11] what is it serving? [13:48:12] either that is the one serving lang.project.beta.wmflabs.org [13:48:25] or it s one with Jessie [13:48:34] cause the Varnish caches on beta currently uses Trusty [13:48:43] and we want to migrate to Jessie to catch up with production [13:48:59] cache-text02 is serving things [13:49:03] cache-text03 has jessie [13:49:13] Is this https://phabricator.wikimedia.org/T98758 ? [13:49:43] so cache-text03 is Jessie [13:49:51] and yeah that is the bug :-} [13:50:13] 10Deployment-Systems, 6operations, 5Patch-For-Review: Trebuchet doesn't like when a deployer server is also a minion, a edge case for scap - https://phabricator.wikimedia.org/T67549#1488080 (10fgiunchedi) 5Open>3Resolved a:3fgiunchedi >>! In T67549#1469568, @thcipriani wrote: > @fgiunchedi works as exp... [13:50:22] 10Beta-Cluster, 10Traffic, 6operations: Upgrade beta-cluster caches to jessie - https://phabricator.wikimedia.org/T98758#1488083 (10hashar) deployment-cache-text03 has been created with Jessie system. That is to prepare the migration of the Trusty cache deployment-cache-text02. [13:50:35] Krenair: we talked a bit about the migration yesterday [13:50:42] 10Beta-Cluster, 10Traffic, 6operations: Upgrade beta-cluster caches to jessie - https://phabricator.wikimedia.org/T98758#1488084 (10Krenair) [13:50:51] since ostriches / thcipriani|afk will handle the migration of Varnishes to Jessie [13:51:01] since they got past experience setting up the staging cluster [13:51:11] should be straight forward, but there is a lot of hardcoded IP everywhere [13:51:12] they should probably assign it or something [13:51:18] I looked at this bug yesterday [13:51:20] and they will probably have to refactor a bunch of things [13:51:36] it is unassigned for now [13:51:43] cause nobody is actively working on it [13:51:47] You'd cookie licked it 6 weeks ago and then left it [13:52:16] I really just triaged it [13:52:20] but no assignee, so I went to see what cache instances we had already before creating one for this task [13:52:26] we talked about it for a few RelEng meeting [13:52:34] but nobody has time to allocate to it right now [13:52:43] So you're not actually working on it [13:52:55] exactly [13:53:33] but eventually ostriches / thcipriani|afk will work on it? [13:53:37] maybe [13:53:46] soonish ™ :-D [13:54:00] we triaged it as high priority yesterday [13:54:13] will probably talk about it during our internal team meeting tonight [13:58:37] 6Release-Engineering, 6operations, 7Database: Audit all existing code to ensure that any extension currently or previously adding blobs to ES has been registering a reference in the text table (and fix up if wrong) - https://phabricator.wikimedia.org/T106388#1488102 (10matthiasmullie) AIUI: the immediate spa... [14:20:37] 6Release-Engineering: Setup Phabricator mirroring from iOS GitHub repo - https://phabricator.wikimedia.org/T107153#1488132 (10BGerstle-WMF) 3NEW a:3mmodell [14:44:02] 6Release-Engineering, 10Gitblit-Deprecate, 6Phabricator, 10Wikimedia-Git-or-Gerrit: Set Git configuration on all Phab repositories - https://phabricator.wikimedia.org/T107156#1488171 (10demon) 3NEW [14:45:03] 6Release-Engineering, 10Gitblit-Deprecate, 6Phabricator, 10Wikimedia-Git-or-Gerrit: Set Git configuration on all Phab repositories - https://phabricator.wikimedia.org/T107156#1488180 (10mmodell) @demon: is it possible, perhaps, to just set these in /etc/gitconfig? [14:47:53] 6Release-Engineering: Setup Phabricator mirroring from iOS GitHub repo - https://phabricator.wikimedia.org/T107153#1488186 (10mmodell) 5Open>3Resolved [15:01:18] 6Release-Engineering, 10Gitblit-Deprecate, 6Phabricator, 10Wikimedia-Git-or-Gerrit: Set Git configuration on all Phab repositories - https://phabricator.wikimedia.org/T107156#1488212 (10demon) Oooh, didn't think of that. Probably yeah :) [15:24:14] 10Continuous-Integration-Infrastructure, 6operations: Phase out lanthanum.eqiad.wmnet - https://phabricator.wikimedia.org/T86658#1488270 (10Cmjohnson) [15:28:07] 10Continuous-Integration-Infrastructure, 6operations: Upload new Zuul .deb package on apt.wikimedia.org for precise-wikimedia - https://phabricator.wikimedia.org/T106499#1488278 (10hashar) 5stalled>3Open [15:29:13] 10Continuous-Integration-Infrastructure, 6operations: Upload new Zuul .deb package on apt.wikimedia.org for precise-wikimedia - https://phabricator.wikimedia.org/T106499#1470255 (10hashar) Bumped the package to wmf3: ``` zuul (2.0.0-327-g3ebedde-wmf3precise1) precise-wikimedia; urgency=medium * 0008-Revert-... [15:32:23] (03CR) 10JanZerebecki: "How about adding that to the defaults? Or are there jobs that should be able to create multiple concurrent runs on nodes?" [integration/config] - 10https://gerrit.wikimedia.org/r/227234 (owner: 10Hashar) [15:40:36] 5Continuous-Integration-Isolation, 6operations: Reinstall labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T107158#1488303 (10hashar) 3NEW [15:40:51] 5Continuous-Integration-Isolation, 6operations: Reinstall labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T107158#1488310 (10hashar) 5Open>3stalled Stalled for now. [15:43:39] 5Continuous-Integration-Isolation, 6operations, 7Blocked-on-Operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1488315 (10hashar) [15:43:58] 5Continuous-Integration-Isolation, 6operations: Reinstall labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T107158#1488316 (10chasemp) >>! In T107158#1488310, @hashar wrote: > Stalled for now. is this a `hashar is going on vacation` stall? :) [15:44:55] 5Continuous-Integration-Isolation, 6operations: Reinstall labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T107158#1488319 (10hashar) Potentially vacations will be a blocker. I have too look at the Debian packages available in apt.wikimedia.org since I think I have manually installed some ;-( [15:45:33] 5Continuous-Integration-Isolation, 6operations, 7Blocked-on-Operations: Backport python-os-client-config 1.3.0-1 from Debian Sid to jessie-wikimedia - https://phabricator.wikimedia.org/T104967#1433420 (10hashar) [15:47:20] greg-g: https://stashbot.wmflabs.org/#/dashboard/elasticsearch/bash [15:47:51] * bd808 cackles evilly [15:49:12] bd808: are they imported from bugzilla quips ? :D [15:49:32] yes. I got them from https://www.mediawiki.org/wiki/Quips [15:50:09] and now you can !bash something awesome here in any channel that stashbot_ is idling in to add a new one [15:50:51] Soon I will make a tool at https://tools.wmflabs.org/bash/ to show them randomly [15:51:37] !bash and now you can !bash something awesome here in any channel that stashbot_ is idling in to add a new one. REF: https://stashbot.wmflabs.org/#/dashboard/elasticsearch/bash [15:53:19] neato! [15:54:55] is there anything logstash *can't* do?! [15:59:50] can I direct link to specific log line? [16:00:08] I will respond to all feature requests with this as higher priority: we should rename icinga.wm.org to christmastree.wm.org [16:07:47] chasemp: each log event has a unique id, so yes you can. [16:13:36] chasemp: I'd support that rename if it plays Christmas music only [16:37:58] 10Continuous-Integration-Infrastructure, 10Gerrit-Migration, 3releng-201516-q1: Prototype CI integration with Differential - https://phabricator.wikimedia.org/T103127#1488549 (10mmodell) a:3mmodell [16:38:33] 10Continuous-Integration-Infrastructure, 10Gerrit-Migration, 3releng-201516-q1: Prototype CI integration with Differential - https://phabricator.wikimedia.org/T103127#1382950 (10mmodell) p:5Normal>3High This is a quarterly goal, therefore, priority -> high [16:39:28] 10Continuous-Integration-Infrastructure, 10Gerrit-Migration, 3releng-201516-q1: Prototype CI integration with Differential - https://phabricator.wikimedia.org/T103127#1488558 (10mmodell) [16:45:06] 5Continuous-Integration-Isolation, 6operations: Reinstall labnodepool1001.eqiad.wmnet - https://phabricator.wikimedia.org/T107158#1488574 (10hashar) Will reinstall it with @andrew on Wednesday 29th. Gotta look at it tonight to figure out which .deb package might be missing, and prepare them for upload on apt.... [16:46:53] 10Continuous-Integration-Infrastructure, 10Gerrit-Migration, 3releng-201516-q1: Prototype CI integration with Differential - https://phabricator.wikimedia.org/T103127#1488585 (10mmodell) Between Herald and Harbormaster, we now have all the pieces in place. Interesting and useful upstream development: [[ htt... [16:47:41] 10Beta-Cluster, 10Traffic, 6operations: Upgrade beta-cluster caches to jessie - https://phabricator.wikimedia.org/T98758#1488592 (10demon) a:3demon [16:49:38] 10Beta-Cluster, 6Labs, 10Wikimedia-Logstash, 5Patch-For-Review: Logstash on beta yields 500 due to NFS outage (can't open /data/project/logstash/.htpasswd) - https://phabricator.wikimedia.org/T102962#1488597 (10bd808) [16:54:42] (03CR) 10Hashar: [C: 031] "Merge whenever, not going to cause problems with mediawiki_selenium < 1.5" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226651 (https://phabricator.wikimedia.org/T104583) (owner: 10Dduvall) [16:55:36] (03CR) 10Hashar: [C: 031] "Go go go !" [selenium] - 10https://gerrit.wikimedia.org/r/226653 (owner: 10Dduvall) [16:56:11] 10Beta-Cluster, 10Wikimedia-Logstash: Make beta logstash server based on a Trusty base image - https://phabricator.wikimedia.org/T78195#1488617 (10bd808) 5Open>3Resolved a:3bd808 I actually jumped over trusty to jessie: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-logstash2.deployment-pre... [16:58:32] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1488623 (10hashar) CI expects them to be prefixed with `wmf/`. That is to filter them out in Zuul ex: ``` zuul/layout.yaml: branch: ^(?!wmf/.*)$ zuul/layout.yaml: bra... [17:25:08] 6Release-Engineering: Setup Phabricator mirroring from iOS GitHub repo - https://phabricator.wikimedia.org/T107153#1488677 (10mmodell) [17:25:17] PROBLEM - Puppet failure on nodepool-t105406 is CRITICAL 100.00% of data above the critical threshold [0.0] [17:34:09] 10Deployment-Systems: Adopt Semantic Versioning format for WMF deploy branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T67306#1488719 (10mmodell) @hashar: I won't be removing the wmf/ prefix [17:34:53] (03PS5) 10Paladox: Update HitCounters tests [integration/config] - 10https://gerrit.wikimedia.org/r/227438 [17:36:22] (03PS3) 10Paladox: Add BlogPage to testextension [integration/config] - 10https://gerrit.wikimedia.org/r/227217 [17:36:54] (03PS4) 10Paladox: Update BlueSky tests [integration/config] - 10https://gerrit.wikimedia.org/r/226635 [17:37:13] (03PS9) 10Paladox: Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 [17:38:42] (03CR) 10Paladox: [C: 031] Add jenkings test for BoilerPlate [integration/config] - 10https://gerrit.wikimedia.org/r/226680 (owner: 10Paladox) [17:39:16] 6Release-Engineering, 6operations, 7Mobile: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1488745 (10MaxSem) 3NEW [17:40:18] (03PS4) 10Paladox: Update WikidataPageBanner tests [integration/config] - 10https://gerrit.wikimedia.org/r/226913 [17:47:43] 6Release-Engineering, 6operations, 7Mobile: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1488778 (10MaxSem) [18:39:57] (03PS3) 10JanZerebecki: zuul: function for OFFLINE_NODE_WHEN_COMPLETE [integration/config] - 10https://gerrit.wikimedia.org/r/220149 (https://phabricator.wikimedia.org/T103551) (owner: 10Hashar) [18:41:59] (03CR) 10JanZerebecki: [C: 032] zuul: function for OFFLINE_NODE_WHEN_COMPLETE [integration/config] - 10https://gerrit.wikimedia.org/r/220149 (https://phabricator.wikimedia.org/T103551) (owner: 10Hashar) [18:43:42] (03Merged) 10jenkins-bot: zuul: function for OFFLINE_NODE_WHEN_COMPLETE [integration/config] - 10https://gerrit.wikimedia.org/r/220149 (https://phabricator.wikimedia.org/T103551) (owner: 10Hashar) [18:44:15] (03PS2) 10JanZerebecki: Publish phpunit coverage for utfnormal, at-ease & AhoCorasick [integration/config] - 10https://gerrit.wikimedia.org/r/220937 (owner: 10Legoktm) [18:49:50] Train deploy delayed? [18:51:18] twentyafterfour: ^ :) [18:51:42] James_F: maybe, mukunda's trying to switch us to semver :) [18:51:51] Oh, wait, trying that /now/? [18:51:56] It'll break ForrestBot. [18:52:15] I didn't realise it was so urgent. [18:54:59] 10Deployment-Systems, 10ReleaseTaggerBot: Update ReleaseTaggerBot to deal with SemVer for WMF deployed branches (eg 1.23.0-wmf.6) - https://phabricator.wikimedia.org/T107192#1489218 (10greg) 3NEW [19:01:05] greg-g: Thanks. :-) [19:04:22] (03CR) 10JanZerebecki: "Test run AhoCorasick: https://integration.wikimedia.org/ci/job/phpunit-coverage-publish/10/console -> https://integration.wikimedia.org/co" [integration/config] - 10https://gerrit.wikimedia.org/r/220937 (owner: 10Legoktm) [19:08:59] (03CR) 10JanZerebecki: "at-ease: https://integration.wikimedia.org/ci/job/phpunit-coverage-publish/11/console -> https://integration.wikimedia.org/cover/at-ease/" [integration/config] - 10https://gerrit.wikimedia.org/r/220937 (owner: 10Legoktm) [19:18:19] (03CR) 10JanZerebecki: "utfnormal: https://integration.wikimedia.org/ci/job/phpunit-coverage-publish/12/console -> https://integration.wikimedia.org/cover/utfnorm" [integration/config] - 10https://gerrit.wikimedia.org/r/220937 (owner: 10Legoktm) [19:18:30] (03CR) 10JanZerebecki: [C: 032] Publish phpunit coverage for utfnormal, at-ease & AhoCorasick [integration/config] - 10https://gerrit.wikimedia.org/r/220937 (owner: 10Legoktm) [19:19:58] (03Merged) 10jenkins-bot: Publish phpunit coverage for utfnormal, at-ease & AhoCorasick [integration/config] - 10https://gerrit.wikimedia.org/r/220937 (owner: 10Legoktm) [19:23:40] (03CR) 10JanZerebecki: [V: 04-1] "Needs manual rebase." [integration/config] - 10https://gerrit.wikimedia.org/r/173830 (owner: 10Hashar) [19:29:31] (03PS2) 10JanZerebecki: Add QuickSearchLookup generic basic tests [integration/config] - 10https://gerrit.wikimedia.org/r/227341 (owner: 10Florianschmidtwelzow) [19:31:29] James_F: should I wait until next week? I don't want to break too much stuff [19:31:38] twentyafterfour: Possibly? [19:31:47] might be wise [19:31:53] (03CR) 10JanZerebecki: [C: 032] Add QuickSearchLookup generic basic tests [integration/config] - 10https://gerrit.wikimedia.org/r/227341 (owner: 10Florianschmidtwelzow) [19:33:24] (03Merged) 10jenkins-bot: Add QuickSearchLookup generic basic tests [integration/config] - 10https://gerrit.wikimedia.org/r/227341 (owner: 10Florianschmidtwelzow) [19:34:15] well I'm gonna push wmf/1.26.0-wmf.16 either way, but I can also make a wmf/1.26wmf16 ... would be good to sanity check that they are equivalent [19:34:35] * James_F nods. [19:35:54] PROBLEM - Puppet failure on integration-slave-trusty-1014 is CRITICAL 40.00% of data above the critical threshold [0.0] [19:36:42] (03PS1) 10Hashar: Merge branch 'debian/precise-wikimedia' into debian/trusty-wikimedia [integration/zuul] (debian/trusty-wikimedia) - 10https://gerrit.wikimedia.org/r/227510 [19:37:11] (03PS2) 10JanZerebecki: Add composer test to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/226587 (owner: 10Florianschmidtwelzow) [19:39:17] PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL 33.33% of data above the critical threshold [0.0] [19:39:59] twentyafterfour: what would you need to know about Jenkins to hook harbormaster with it ? [19:40:13] just a way to trigger build / retrieve the result rights ? [19:40:20] (03CR) 10JanZerebecki: [C: 032] Add composer test to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/226587 (owner: 10Florianschmidtwelzow) [19:42:03] (03Merged) 10jenkins-bot: Add composer test to GoogleLogin [integration/config] - 10https://gerrit.wikimedia.org/r/226587 (owner: 10Florianschmidtwelzow) [19:44:03] thcipriani: zuul_2.0.0-327-g3ebedde-wmf3trusty1_amd64.deb ! [19:44:12] I think I have a good workflow I can actually document now [19:45:03] PROBLEM - Puppet failure on deployment-mediawiki01 is CRITICAL 44.44% of data above the critical threshold [0.0] [19:46:40] (03CR) 10JanZerebecki: [V: 04-1] "Needs manual rebase." [integration/config] - 10https://gerrit.wikimedia.org/r/208893 (owner: 10Legoktm) [19:47:32] hashar: yeah just to trigger a build and get the result [19:47:44] I almost got it figured out but it needs some kind of authentication [19:47:57] legoktm: if around, would be nice to have your https://www.mediawiki.org/wiki/User:Legoktm/ci hosted under integration.wm.o :) [19:48:13] legoktm: greg been asking for some metrics recently and I think that would be a great start [19:48:36] yeah, that page is killer [19:48:37] twentyafterfour: definitely, not that hard. I am a core maintainer for a python module (python-jenkins) [19:48:57] twentyafterfour: so most of the trouble has been dealt with already. Just gotta extract bits from tthe module [19:49:23] !log reloading zuul b1b2cab..b02830e [19:49:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [19:49:34] hashar: phabricator has the ability to call out to a http url (get or post) without any code at all [19:51:46] PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL 20.00% of data above the critical threshold [0.0] [19:54:30] PROBLEM - Puppet failure on deployment-mediawiki02 is CRITICAL 30.00% of data above the critical threshold [0.0] [19:54:52] hashar, greg-g: yeah, making that a proper report is on my list of things to do. it's doing lots of ugly stuff like attempting to parse the zuul debug log output that it shouldn't. the code is in https://github.com/legoktm/tools-ci though [19:55:05] GH! [19:55:14] :P [19:55:22] hah " tmp until moved to gerrit [19:55:22] " [19:56:09] legoktm: noticed and I think I sent a lame PR [19:56:21] ...which I totally missed :( [19:56:26] twentyafterfour: or you can trigger the jobs via the Gearman server :D [19:56:52] PROBLEM - Puppet failure on integration-slave-trusty-1012 is CRITICAL 40.00% of data above the critical threshold [0.0] [19:56:58] legoktm: we will probably want to have the dashboard to track our support release branches as well :/ [19:57:15] PROBLEM - Puppet failure on integration-slave-trusty-1013 is CRITICAL 44.44% of data above the critical threshold [0.0] [19:57:19] yeah. I mainly built this for my own usage and James_F [19:57:23] grr puppet [20:00:35] 10Continuous-Integration-Infrastructure, 6operations: Upload new Zuul .deb package on apt.wikimedia.org for precise-wikimedia and trusty-wikimedia - https://phabricator.wikimedia.org/T106499#1489468 (10hashar) [20:01:01] 10Continuous-Integration-Infrastructure, 6operations: Upload new Zuul .deb package on apt.wikimedia.org for precise-wikimedia and trusty-wikimedia - https://phabricator.wikimedia.org/T106499#1470255 (10hashar) Finally rebuild the package for Trusty as zuul_2.0.0-327-g3ebedde-wmf3trusty1 . I have updated the ta... [20:01:56] PROBLEM - Puppet failure on deployment-videoscaler01 is CRITICAL 60.00% of data above the critical threshold [0.0] [20:11:41] jzerebecki: you might find gallium:/home/legoktm/test_extension.py helpful, it automatically triggers the 'test' pipeline for the last change in a repo so you don't have to comment 'recheck' [20:15:53] RECOVERY - Puppet failure on integration-slave-trusty-1014 is OK Less than 1.00% above the threshold [0.0] [20:18:40] what the... legoktm is using Flow as a bug tracker? https://www.mediawiki.org/wiki/User_talk:Legoktm/ci [20:22:23] hehe. that's what it is, right? [20:24:07] legoktm: thx. test_extension.py is what I need. [20:25:10] it also shows me how to trigger some of the stuff that recheck can't like a post merge publish job [20:30:08] greg-g: It works well. :-) [20:32:15] RECOVERY - Puppet failure on integration-slave-trusty-1013 is OK Less than 1.00% above the threshold [0.0] [20:38:11] 10Deployment-Systems, 6Release-Engineering, 6operations: Corrupt /srv/deployment/scap/scap checkouts on WMF prod cluster - https://phabricator.wikimedia.org/T103441#1489642 (10greg) [20:38:15] 6Release-Engineering, 10Wikidata, 10Wikimedia-General-or-Unknown, 6operations: Wikidata and Wikiversity logo 404ing on wikimedia.org - https://phabricator.wikimedia.org/T103296#1489644 (10greg) [20:38:19] 6Release-Engineering, 6Labs, 6operations, 10wikitech.wikimedia.org, 5Patch-For-Review: silver / scap - Could not get latest version: 403 Forbidden - https://phabricator.wikimedia.org/T103138#1489646 (10greg) [20:42:48] 6Release-Engineering: Testing: where does it hurt? - https://phabricator.wikimedia.org/T106600#1489691 (10greg) 5Open>3Resolved good work, team [21:01:23] !log upgraded nutcracker on mediawiki01 [21:01:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:03:59] !log upgraded nutcracker on mediawiki02 [21:04:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:05:56] !log upgraded nutcracker on mediawiki03 [21:05:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:06:54] RECOVERY - Puppet failure on integration-slave-trusty-1012 is OK Less than 1.00% above the threshold [0.0] [21:21:44] 6Release-Engineering: Testing: where does it hurt? - https://phabricator.wikimedia.org/T106600#1489869 (10hashar) Thanks that was a good first session. [21:23:35] (03CR) 10Dduvall: [C: 032] Configure video capture for headless mw-selenium jobs [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226651 (https://phabricator.wikimedia.org/T104583) (owner: 10Dduvall) [21:24:07] (03Merged) 10jenkins-bot: Configure video capture for headless mw-selenium jobs [integration/jenkins] - 10https://gerrit.wikimedia.org/r/226651 (https://phabricator.wikimedia.org/T104583) (owner: 10Dduvall) [21:26:27] (03CR) 10Dduvall: [C: 032] Release minor version 1.5.0 [selenium] - 10https://gerrit.wikimedia.org/r/226653 (owner: 10Dduvall) [21:27:31] (03Merged) 10jenkins-bot: Release minor version 1.5.0 [selenium] - 10https://gerrit.wikimedia.org/r/226653 (owner: 10Dduvall) [21:33:43] (03PS1) 10Dduvall: Test video recording of mw-selenium failures [selenium] - 10https://gerrit.wikimedia.org/r/227583 [21:34:58] (03CR) 10jenkins-bot: [V: 04-1] Test video recording of mw-selenium failures [selenium] - 10https://gerrit.wikimedia.org/r/227583 (owner: 10Dduvall) [21:37:24] 6Release-Engineering, 6operations, 7Mobile: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489889 (10hashar) So my questions are: what the hell is that code base for? was it a one off experiment? is that actually receiving traffic? can we shoot it? which tea... [21:39:16] (03Abandoned) 10Dduvall: Test video recording of mw-selenium failures [selenium] - 10https://gerrit.wikimedia.org/r/227583 (owner: 10Dduvall) [21:41:25] 6Release-Engineering, 6operations, 7Mobile: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489893 (10demon) >>! In T107172#1489889, @hashar wrote: > So my questions are: > > what the hell is that code base for? Firefox OS, lol. > was it a one off experimen... [21:43:02] 6Release-Engineering, 6operations, 7Mobile: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489895 (10greg) @brion tell us we can kill the WikipediaMobileFirefoxOS thingy, please? [21:47:33] 6Release-Engineering, 6operations, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489899 (10hashar) [21:48:35] 6Release-Engineering, 6operations, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489902 (10dr0ptp4kt) The app is part of the Partnerships portfolio. It's in maintenance / bugfix mode. [21:50:09] 6Release-Engineering, 6operations, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489909 (10greg) >>! In T107172#1489902, @dr0ptp4kt wrote: > The app is part of the Partnerships portfolio. It's in maintenance / bugfix mode. Adam:... [21:52:17] have a good afternoon [21:54:41] 6Release-Engineering, 6operations, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489922 (10dr0ptp4kt) @greg, FFOS is more targeted at Global South regions, so probably the simplest would be #zero. [21:55:35] 6Release-Engineering, 6Zero, 6operations, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1489923 (10greg) [22:21:15] (03PS1) 10Dduvall: Fix SKIP_TMPFS conditional [integration/jenkins] - 10https://gerrit.wikimedia.org/r/227593 [22:26:51] ostriches: mind just double checking my logic there? ^ [22:27:03] * marxarelli 's brain is boiling in this heat [22:29:12] marxarelli: 94 up here now, I just biked home from coffee shop, still sweating [22:29:24] (sorry for that, twentyafterfour, during our 1:1 ;) ) [22:30:07] greg-g: tell me you had an iced coffee! [22:30:30] :) nope, drip :) [22:30:35] oh man, a cold brew coffee from that red door place sounds good [22:30:41] greg-g: !! [22:31:05] I know, I'm a weirdo. it was nice a chilled in the shop though [22:32:36] we're heading up to Russian River tomorrow night. i imagine it's going to be similar weather [22:37:47] Ooooh, I want an iced coffee now [22:46:37] nice! my neck of the woods :) [22:50:20] greg-g: yeah! i think we're going to do some river floating and hit up stumptown [22:52:41] 6Release-Engineering, 6Zero, 6operations, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1490081 (10greg) Zero team: Can one of you please assist with this request to move the Fireofx OS App out of the mediawiki-config repository?... [22:56:54] (03CR) 10Dduvall: [C: 032] Fix SKIP_TMPFS conditional [integration/jenkins] - 10https://gerrit.wikimedia.org/r/227593 (owner: 10Dduvall) [22:58:43] (03Merged) 10jenkins-bot: Fix SKIP_TMPFS conditional [integration/jenkins] - 10https://gerrit.wikimedia.org/r/227593 (owner: 10Dduvall) [23:28:52] (03PS1) 10Krinkle: Add WrappedString test and publisher jobs [integration/config] - 10https://gerrit.wikimedia.org/r/227615 [23:29:20] 10Browser-Tests, 5Patch-For-Review: Support headless gem's video recording feature for headless Jenkins jobs - https://phabricator.wikimedia.org/T104583#1490202 (10dduvall) 5Open>3Resolved Video recordings are now working! I've updated the docs at https://www.mediawiki.org/wiki/Continuous_integration/Brows... [23:29:27] (03CR) 10Krinkle: [C: 032] Add WrappedString test and publisher jobs [integration/config] - 10https://gerrit.wikimedia.org/r/227615 (owner: 10Krinkle) [23:31:44] PROBLEM - Puppet failure on deployment-cache-text02 is CRITICAL 100.00% of data above the critical threshold [0.0] [23:31:57] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Epic, 7Tracking: [EPIC] trigger browser tests from Gerrit (tracking) - https://phabricator.wikimedia.org/T55697#1490214 (10dduvall) [23:31:59] 10Browser-Tests, 10Continuous-Integration-Infrastructure: Run subset of browser tests on isolated CI instances per commit submitted in mediawiki/core - https://phabricator.wikimedia.org/T54424#1490215 (10dduvall) [23:32:01] 10Browser-Tests, 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Define JJB builder for running a subset of integration MW-Selenium tests - https://phabricator.wikimedia.org/T103039#1490212 (10dduvall) 5Open>3Resolved There's still room for improvement in the way of scenario isolation but the... [23:37:47] (03PS1) 10Dduvall: Use MW-Selenium setup slave script [integration/config] - 10https://gerrit.wikimedia.org/r/227616 [23:43:51] !log running `jenkins-jobs update config/ 'mwext-mw-selenium'` to deploy I7afa07e9f559bffeeebaf7454cc6b39a37e04063 [23:43:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:44:58] (03CR) 10Dduvall: [C: 032] "Successfully deployed and tested." [integration/config] - 10https://gerrit.wikimedia.org/r/227616 (owner: 10Dduvall) [23:47:26] (03CR) 10JanZerebecki: "recheck" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/192177 (owner: 10Legoktm) [23:48:01] (03CR) 10jenkins-bot: [V: 04-1] Add script to create a composer.local.json based on a list of extensions [integration/jenkins] - 10https://gerrit.wikimedia.org/r/192177 (owner: 10Legoktm) [23:51:00] (03Merged) 10jenkins-bot: Add WrappedString test and publisher jobs [integration/config] - 10https://gerrit.wikimedia.org/r/227615 (owner: 10Krinkle) [23:56:54] (03PS1) 10JanZerebecki: Make python tests verbose. [integration/jenkins] - 10https://gerrit.wikimedia.org/r/227619 [23:58:34] (03CR) 10JanZerebecki: [C: 032] Make python tests verbose. [integration/jenkins] - 10https://gerrit.wikimedia.org/r/227619 (owner: 10JanZerebecki)