[01:00:27] RECOVERY - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is OK: OK: Less than 100.00% above the threshold [0.0] [01:39:21] 06Release-Engineering-Team, 06Operations, 10Phabricator: reinstall iridium (phabricator) as phab1001 with jessie - https://phabricator.wikimedia.org/T152129#3246213 (10Dzahn) This will be unblocked after T164810. [02:36:14] (03PS1) 10Aude: Bump Wikidata to wmf/1.30.0-wmf.1 [tools/release] - 10https://gerrit.wikimedia.org/r/352740 [02:51:08] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy, 07Software-Licensing, 15User-zeljkofilipin: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#3246240 (10Mattflaschen-WMF) The footer looks good now, except the license links are broken. mediawiki-ruby-api goes... [06:20:31] Yippee, build fixed! [06:20:32] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #355: 09FIXED in 1 hr 40 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/355/ [06:42:52] (03PS2) 10Hashar: Jobs for mediawiki/skins/Athena [integration/config] - 10https://gerrit.wikimedia.org/r/352166 [06:43:30] Yippee, build fixed! [06:43:30] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #355: 09FIXED in 2 hr 3 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/355/ [06:46:35] (03CR) 10Hashar: [C: 032] Jobs for mediawiki/skins/Athena [integration/config] - 10https://gerrit.wikimedia.org/r/352166 (owner: 10Hashar) [06:47:38] (03Merged) 10jenkins-bot: Jobs for mediawiki/skins/Athena [integration/config] - 10https://gerrit.wikimedia.org/r/352166 (owner: 10Hashar) [08:14:12] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy, 07Software-Licensing, 15User-zeljkofilipin: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#3246547 (10Rammanojpotla) yes I have seen them but they were even proper when I tried them in my master branch after... [08:14:44] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy, 07Software-Licensing, 15User-zeljkofilipin: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#3246548 (10Rammanojpotla) yes I have seen them but they were even proper when I tried them in my master branch after... [08:16:46] sigh the current repo on puppetmaster02 doesn't rebase cleanly on production, elk upgrade patch [08:16:56] I wanted to remove some local cherry picks, now obsolete [08:26:55] hashar :D [08:28:10] addshore: hi [08:28:16] seen your email: I have no clue [08:28:23] okay :( [08:28:29] the unitest tables in mysql are temporary tables [08:28:39] and are for the current connection iirc. [08:28:48] I'm sure I have come accross it before, bit I tried searching my emails and phab and found nothing [08:28:58] if one creates a second database connection, it will not see those tables [08:29:12] oooooh [08:29:12] wait [08:29:23] it might have something to do with that I guess [08:29:40] at least it got cloned: [08:29:45] 4.2199 21.2M CloneDatabase::cloneTableStructure dropping unittest_user_groups [08:29:46] 4.2201 21.2M CloneDatabase::cloneTableStructure duplicating user_groups to unittest_user_groups [08:29:47] (from the debug log https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/4798/artifact/log/mw-debug-cli.log/*view*/ ) [08:29:58] [DBConnection] Wikimedia\Rdbms\LoadBalancer::openConnection: calling initLB() before first connection. [08:29:58] [DBConnection] Connected to database 0 at '127.0.0.1:3306'. [08:29:59] [DBQuery] SQL ERROR: Table 'jenkins_u0_mw.unittest_user_groups' doesn't exist (127.0.0.1:3306) [08:30:35] okay, so maybe it is something to do with using a different connection. Thanks for the possible hint, I'll look into it a bit more now! [08:31:09] addshore: I am not sure what those DBConnectiong debug log are. Maybe it is properly reusing a connection [08:31:47] addshore: and maybe I am wrong and the temporary tables are actually kept :-} [08:33:21] ahh, that was the odd thing, they run locally without the issue :p [08:33:55] ;D [08:34:34] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy, 07Software-Licensing, 15User-zeljkofilipin: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#3246598 (10zeljkofilipin) I am also sure I have tested the links locally and it worked. I will take another look. [08:36:39] addshore: phpunit/MediaWikiTestCase.php refers to user_groups [08:36:47] most probably it deletes the table if it got used somehow? [08:37:19] in resetDb [08:37:59] it looks like that just truncates the table, not drops it [08:38:25] "A TEMPORARY table is visible only to the current connection" [08:38:40] I reckon that has to be it, and somewhere I am getting a new connection [08:38:50] maybe [08:38:56] but " [DBConnection] Connected to database 0 at '127.0.0.1:3306'." [08:39:08] might be issued after it reconnected to the existing connection [08:39:20] oh [08:39:21] $this->tablesUsed[] = 'user_groups'; [08:39:23] in your code [08:39:33] so yeah resetDB() probably drop the table [08:39:36] It does the same thing with or without that [08:39:50] I just stuck that in the last PS to see what would happen [08:39:58] ok [08:47:53] 10Browser-Tests-Infrastructure, 07Documentation, 07Easy, 07Software-Licensing, 15User-zeljkofilipin: Ruby gem documentation should state license - https://phabricator.wikimedia.org/T94001#3246614 (10Rammanojpotla) @zeljkofilipin it is working properly in local repository in the branch which I have pushed... [08:56:01] 10Beta-Cluster-Infrastructure, 10media-storage, 13Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3246620 (10hashar) [09:03:21] 10Beta-Cluster-Infrastructure, 10media-storage, 13Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3246631 (10hashar) The Swift servers have been rebuild to use Jessie. `object-server` workers... [09:05:00] 10Beta-Cluster-Infrastructure, 10media-storage, 13Patch-For-Review: deployment-ms-be03.deployment-prep and deployment-ms-be04.deployment-prep have high load / system CPU - https://phabricator.wikimedia.org/T160990#3246634 (10hashar) [09:46:41] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: For MediaWiki extensions, merge composer test into mwext-textextension / mediawiki-extensions jobs - https://phabricator.wikimedia.org/T161895#3246699 (10hashar) There are a few extensions having a `composer.json` file but that... [10:07:18] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3246791 (10Qse24h) [10:09:17] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3246854 (10Qse24h) [10:09:53] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3245591 (10Qse24h) [10:10:04] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3246876 (10Qse24h) [10:10:11] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3245591 (10Qse24h) [10:10:33] Can someone disable https://phabricator.wikimedia.org/p/Qse24h/ andre__ [10:10:39] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3245591 (10Qse24h) [10:11:05] p858snake, thanks, blocked [10:11:20] urgh. And no clean up, sigh :( [10:12:24] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3245591 (10Qse24h) [10:13:22] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10Qse24h) [10:15:34] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10Qse24h) [10:16:42] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3247065 (10Qse24h) [10:16:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3244205 (10Qse24h) [10:18:17] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10Qse24h) [10:18:38] Looks like this spam is going to go on for a long time. Can wikibugs be stopped and the queue cleared? [10:18:54] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10Qse24h) [10:19:17] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10Qse24h) [10:19:37] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10Qse24h) [10:19:42] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3244205 (10Qse24h) [10:19:53] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3244205 (10Qse24h) [10:19:55] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10Qse24h) [10:20:18] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3244205 (10Qse24h) [10:20:39] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:20:58] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3244205 (10Qse24h) [10:21:34] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3244205 (10Qse24h) [10:21:47] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3244205 (10Qse24h) [10:22:51] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:23:59] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:25:27] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:25:34] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:25:57] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:26:20] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:26:25] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:26:52] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:27:27] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3243426 (10Qse24h) [10:28:01] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:28:03] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:28:37] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:28:41] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:28:45] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:29:15] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:29:51] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3243094 (10Qse24h) [10:32:28] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3247543 (10Aklapper) 05duplicate>03Open [10:32:35] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3247546 (10Aklapper) 05duplicate>03Open [10:33:10] 10Beta-Cluster-Infrastructure: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3247564 (10Aklapper) 05duplicate>03Open [10:33:28] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Run WebdriverIO tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3247572 (10Aklapper) 05duplicate>03Open [10:51:21] hashar: hey, does this config looks okay to you? https://gerrit.wikimedia.org/r/352795 [10:51:33] I wasn't able to get it more cleaned [10:56:22] (03Abandoned) 10Hashar: No more semanticness in prod [tools/release] - 10https://gerrit.wikimedia.org/r/352724 (owner: 10MaxSem) [10:56:39] (03CR) 10Hashar: [C: 031] Stop branching Semantic stuff for wikitech [tools/release] - 10https://gerrit.wikimedia.org/r/352720 (https://phabricator.wikimedia.org/T53642) (owner: 10Reedy) [11:03:58] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:06] (03CR) 10Peachey88: [C: 031] "He said semantic! https://upload.wikimedia.org/wikipedia/mediawiki/6/69/Hesaidsemanticga2.jpg" [tools/release] - 10https://gerrit.wikimedia.org/r/352720 (https://phabricator.wikimedia.org/T53642) (owner: 10Reedy) [11:44:38] 10Gerrit, 10Analytics-Tech-community-metrics: Numerous Gerrit (draft) patchsets cannot be accessed: "Cannot display change because it has no revisions." - https://phabricator.wikimedia.org/T161207#3247722 (10Aklapper) 05Open>03Invalid Alright, thanks everybody. Current behavior looks intentional (as draft... [12:02:32] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.30.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T162954#3247770 (10TTO) [12:19:16] !log Unbroke puppet on deployment-irc and deployment-urldownloader . Both choked on a ruby one-liner, fixed via https://gerrit.wikimedia.org/r/#/c/336840/ [12:19:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:31:45] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [12:40:13] \O/ [12:44:09] !log deployment-ircd upgrading puppet 3.7.2 => 3.8.5 [12:44:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:45:21] Project beta-scap-eqiad build #154595: 04FAILURE in 1 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/154595/ [12:54:18] ^^^ Max retries exceeded with url: /logstash-*/_search [12:55:42] hashar: operations is fixing something related to logstash [12:55:52] Yippee, build fixed! [12:55:53] Project beta-scap-eqiad build #154596: 09FIXED in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/154596/ [12:56:18] 7:50 AM <@moritzm> Invalid relationship: Rsyslog::Conf[etcd] {require => File[/etc/logrotate.d/etcd}, because File doesn't seem to be in the catalogue [12:56:53] Zppix: yeah [12:56:58] the jenkins job is fixed :) [13:03:35] (03CR) 10Aude: [C: 032] Bump Wikidata to wmf/1.30.0-wmf.1 [tools/release] - 10https://gerrit.wikimedia.org/r/352740 (owner: 10Aude) [13:05:10] (03Merged) 10jenkins-bot: Bump Wikidata to wmf/1.30.0-wmf.1 [tools/release] - 10https://gerrit.wikimedia.org/r/352740 (owner: 10Aude) [13:07:22] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [14:03:44] 10Beta-Cluster-Infrastructure, 07Chinese-Sites: Request for adminship on zhwp beta cluster - https://phabricator.wikimedia.org/T164732#3248116 (10Liuxinyu970226) [14:12:48] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:23:03] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3248160 (10Joe) a:03Joe [14:29:36] PROBLEM - Host deployment-phab02 is DOWN: CRITICAL - Host Unreachable (10.68.19.232) [14:41:10] hashar (or anyone else): I'm having more trouble than expected with aligning version of logstash on deployment-prep, it is still not up [14:41:35] what is the right place to communicate that? [14:43:54] wikitech-l? [14:44:13] yeah I would say wikitech-l [14:44:28] that is probably the best place to spa^Wspread a message :} [14:44:29] ok, I'll send an email... [14:47:48] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [15:10:02] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3248381 (10Joe) so, after some digging, I found out that `conf2002.codfw.wmnet` had, for some reason, auth enabled on etcd (while we now just proxy through nginx) and for some reason only had the root user availa... [15:14:43] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: Create "High Priority" gate-and-submit pipeline - https://phabricator.wikimedia.org/T160668#3248408 (10thcipriani) 05Open>03Resolved Saw this in action during European mid-day SWAT. Seems to be working to me. I will continue... [15:18:11] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: Create "High Priority" gate-and-submit pipeline - https://phabricator.wikimedia.org/T160668#3248421 (10hashar) Yup looks good to me after the European SWAT. Kudos on the config :-} [15:26:11] 10Gerrit, 06Release-Engineering-Team, 06Operations: replace gerrit server (ytterbium) with jessie server (lead) - https://phabricator.wikimedia.org/T125018#3248449 (10Cmjohnson) [15:36:14] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Raise priority for operations-mw-config-composer-hhvm-jessie from the gate-and-submit pipeline - https://phabricator.wikimedia.org/T162076#3248473 (10hashar) 05Open>03Resolved a:03hashar It is already at the highest priority and in a standalone... [15:48:28] 10Continuous-Integration-Infrastructure (Little Steps Sprint): Raise priority for operations-mw-config-composer-hhvm-jessie from the gate-and-submit pipeline - https://phabricator.wikimedia.org/T162076#3151697 (10thcipriani) >>! In T162076#3152132, @hashar wrote: > Probably related to {T160668} This one was onl... [16:06:33] (03PS2) 10Hashar: Allow composer-test to run in different directory [integration/config] - 10https://gerrit.wikimedia.org/r/352128 (https://phabricator.wikimedia.org/T161895) [16:15:25] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3248701 (10EBernhardson) I'm not really sure what has to be done to switch logstash-plugins from trebuchet to scap, but basically the only thing it needs to... [17:22:29] 06Release-Engineering-Team (Deployment-Blockers), 05MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)), 05Release: MW-1.30.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T162954#3248813 (10mmodell) p:05Normal>03High [18:05:18] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3249016 (10Umherirrender) I have run composer validate locally on all extensions... [18:08:57] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mass adding of jakub-onderka/php-console-highlighter missed changes to composer.lock files - https://phabricator.wikimedia.org/T164751#3249017 (10Umherirrender) For information: DonationInterface is the last repo wi... [18:24:51] 10Scap, 10Parsoid, 15User-Joe: Parsoid deploy failed - https://phabricator.wikimedia.org/T164793#3249068 (10mobrovac) 05Open>03Resolved p:05Triage>03Normal Parsoid has been successfully deployed now, so I'm declaring this resolved. Thnx @Joe for looking into it and fixing! [18:40:37] 06Release-Engineering-Team (Deployment-Blockers), 05MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)), 05Release: MW-1.30.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T162954#3249123 (10mmodell) [19:19:54] 10Continuous-Integration-Infrastructure (Little Steps Sprint), 13Patch-For-Review: For MediaWiki extensions, merge composer test into mwext-textextension / mediawiki-extensions jobs - https://phabricator.wikimedia.org/T161895#3146533 (10Umherirrender) For all extensions from above now exists patch sets, also f... [19:52:34] (03PS2) 10Umherirrender: Add some composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/352347 [19:52:46] (03PS3) 10Umherirrender: Add some composer jobs [integration/config] - 10https://gerrit.wikimedia.org/r/352347 [19:55:03] (03CR) 10Umherirrender: "This patch set now contains all tests necessary, when I82a2e9b4f22acbc9efa517b5148917b99c244b10 is not merged or it will be delayed and th" [integration/config] - 10https://gerrit.wikimedia.org/r/352347 (owner: 10Umherirrender) [20:29:25] (03PS1) 10Chad: Make-release: Drop stupid remote update code path [tools/release] - 10https://gerrit.wikimedia.org/r/352894 [20:30:39] greg-g: hey, can I get another window tomorrow for https://phabricator.wikimedia.org/T159753 [20:32:25] three-hour-long main. script [20:33:12] (03PS1) 10Chad: make-release: Remove --gitroot, it's dumb [tools/release] - 10https://gerrit.wikimedia.org/r/352896 [20:39:49] 06Release-Engineering-Team (Deployment-Blockers), 05MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)), 05Release: MW-1.30.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T162954#3249600 (10mmodell) [20:43:42] (03PS1) 10Chad: make-release: Collapse the different versions into 1.23+ [tools/release] - 10https://gerrit.wikimedia.org/r/352904 [20:44:20] 06Release-Engineering-Team (Deployment-Blockers), 05MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)), 05Release: MW-1.30.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T162954#3249628 (10mmodell) [20:47:17] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3249647 (10thcipriani) **tl;dr**: The timestamps of when the error-rate spiked on the canary servers and when the deplo... [20:53:59] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3244333 (10mmodell) there may be some delay in getting events into logstash but I guess that shouldn't screw up the tim... [20:54:56] RainbowSprinkles: mind helping me debug something relating to an earlier deployment? that I thought the train would fix? [20:55:33] (or anyone else relangy for that matter) [20:56:25] undeploy all the things [20:56:36] or Reedy you might be able to know! :p [20:56:38] greg-g: Added https://wikitech.wikimedia.org/w/index.php?title=Deployments&diff=1758884&oldid=1758879. Feel free to remove at any time [20:57:08] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3249668 (10mmodell) cc @bd808 for logstash clue train {icon train} . [20:57:48] basically we deployed https://gerrit.wikimedia.org/r/#/c/352817 earlier witha full scap and some message in JS are missing :/ [20:58:27] addshore: Don't you need to list the messages in the RL defintion? [20:58:47] Are all of them there? [20:59:18] *looks* [21:00:15] Reedy: yup, looks like that is the sisue [21:00:16] *issue [21:00:20] stupid master being broken [21:00:23] heh [21:00:29] Shouldn't need a scap then [21:00:34] Just fix extension.json and sync [21:02:07] bah, cool, will do, silly master [21:02:41] thanks for spotting! [21:04:14] Amir1: cool, thanks [21:06:44] hmmm, Reedy https://gerrit.wikimedia.org/r/#/c/352817/1/extension.json on line 118 the message is there but doesnt appear on prod [21:07:11] infact, all the messages are in the RL definitions [21:08:26] Check the combined l10n json files etc to see if the messages are there? [21:09:22] Reedy: where are they? :P [21:09:53] /srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10 [21:09:57] n [21:10:05] /srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n/upstream [21:11:00] They look to be [21:11:07] Could be some RL cache bug? [21:11:53] mhhhm, any ideas how to unbug it? [21:12:32] reedy@tin:/srv/mediawiki-staging/php-1.29.0-wmf.21/cache/l10n/upstream$ mwscript eval.php enwiki [21:12:33] > echo wfMessage( 'twoColConflict-help-dialog-slide5' )->text(); [21:12:33] Copy and paste the desired parts of the other version into the text in the editor. Only what is in the editor will be published, when you click the "Publish changes" button. [21:12:38] 10Scap: scap did not catch `Notice: Undefined variable: wmgRelatedArticlesShowInSidebar in /srv/mediawiki/wmf-config/CommonSettings.php on line 2893` - https://phabricator.wikimedia.org/T164754#3249748 (10thcipriani) >>! In T164754#3249660, @mmodell wrote: > there may be some delay in getting events into logstas... [21:12:40] That proves it's making it into PHP land [21:12:55] Reedy: yup, all the php messages from the patch turned up just fine [21:13:46] How long since it was deployed? [21:13:53] hours *checks* [21:14:12] first scap finished at 14:10 [21:14:20] 14:10 hashar@tin: Finished scap: TwoColConflict update (duration: 19m 30s) [21:16:16] Might be best asking Roan/Timo if it's RL/js stuff [21:17:03] RoanKattouw ^^ around at all? [21:19:14] Or Krinkle ^^ [21:19:31] I've got a feeling Krinkle is travelling.. soon [21:19:34] Not sure exactly when [21:23:13] mhhhm, silly RL [21:27:59] 10Scap, 13Patch-For-Review: Scap's canary check should stop us from deploying config changes that cause floods of "undefined variable" errors - https://phabricator.wikimedia.org/T162974#3249767 (10thcipriani) 05Open>03Resolved a:03thcipriani This change was deployed and is live now. Thanks for the swat @... [21:31:38] 06Release-Engineering-Team (Deployment-Blockers), 05MW-1.30-release-notes (WMF-deploy-2017-05-09_(1.30.0-wmf.1)), 05Release: MW-1.30.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T162954#3249779 (10mmodell) [21:32:11] yeh, Reedy, this is odd, as it is even happening on test... which is on the new branch [21:32:19] o_0 [21:32:19] I think I'm still missing something [21:35:33] message: 'twoColConflict-help-dialog-slide5', [21:35:39] Where do these go through mw.msg() etc? [21:36:11] https://github.com/wikimedia/mediawiki-extensions-TwoColConflict/blob/wmf/1.30.0-wmf.1/modules/ext.TwoColConflict.HelpDialog.js#L39 [21:36:28] that's not mw.msg [21:36:31] that's just using the key? [21:36:39] https://github.com/wikimedia/mediawiki-extensions-TwoColConflict/blob/wmf/1.30.0-wmf.1/modules/ext.TwoColConflict.HelpDialog.js#L80 [21:37:22] isn't it mw.msg? [21:37:25] not mw.message? [21:37:43] hmm, well, its working for the other 4 messages [21:37:45] hmm [21:37:54] phpstorm reckons mw.message only comes in mmv and popups [21:39:17] but that could be just inference [21:39:38] yeah [21:39:38] https://www.mediawiki.org/wiki/Manual:Messages_API#mw.msg [21:40:09] You think changing to .msg will fix this? :P [21:40:34] nope [21:41:12] could it be something to do with the fact that 1st off only the dir was scaped and then a full scap after? [21:41:34] the messages are in php land [21:41:40] it's nothing to do with scap at this point [21:41:52] if they weren't available in PHP, we would need to look at scap [21:42:03] okay, because inbetween the 2 scaps we got warnings for the messages not being there https://logstash.wikimedia.org/goto/a80533917ba11cd165804e0651525f74 [21:43:17] mhhm, so odd, right i have to leave the office else I'll miss my last train! I'll be back in a bit to continue trying to figure this out [22:01:20] Reedy: addshore: I just arrived in the UK, was traveling [22:01:27] :) [22:02:39] iskrinkleintheuk.co.uk [22:09:35] Reedy: addshore: I'll be able to investigate in an hour or two after dinner. Can you share some links to the issue or commit you deployed and what went wrong and how it was fixed and what you observed in the process? I suspect it might be due to wrong deployment order or something. The resourceloader http cache (varnish) is self-correcting within 5 minutes [22:09:35] for race conditions, and the message cache should never be stale as far as I can tell. We spent quite a bit of effort on this past year to make it more stable. [22:09:52] There's been full scaps for it [22:09:58] When it was deployed (during swat) and of the train after [22:10:12] https://gerrit.wikimedia.org/r/#/c/352817 is the commit [22:10:43] https://labtestwikitech.wikimedia.org/w/index.php?title=Main_Page&type=revision&diff=28393&oldid=2 [22:10:44] 2 full scaps for the deploy of the patch, and the scap for the train since [22:10:46] wrong channel [22:12:34] And no l10nupdate in between or since? [22:13:04] shouldn't have been, no [22:14:52] So messages changed + introduced + translations therefor + usage therefor in one change [22:14:59] and deployed with a full scap [22:15:12] thereof* [22:15:15] should be fine [22:15:24] "full scaps", plural? [22:15:26] bbl [22:40:15] Krinkle: I put everything in https://etherpad.wikimedia.org/p/twocolconflictdeploy [22:42:55] I'm heading to bed but if you find anything feel free to give me a ping :) [22:43:32] I was going to be doing https://gerrit.wikimedia.org/r/#/c/350847/ too in the swat but didnt because of the message issue [22:54:13] (03PS3) 10Reedy: Stop branching Semantic stuff for wikitech [tools/release] - 10https://gerrit.wikimedia.org/r/352720 (https://phabricator.wikimedia.org/T53642) [22:54:31] (03CR) 10Reedy: [C: 032] Stop branching Semantic stuff for wikitech [tools/release] - 10https://gerrit.wikimedia.org/r/352720 (https://phabricator.wikimedia.org/T53642) (owner: 10Reedy) [22:56:10] (03Merged) 10jenkins-bot: Stop branching Semantic stuff for wikitech [tools/release] - 10https://gerrit.wikimedia.org/r/352720 (https://phabricator.wikimedia.org/T53642) (owner: 10Reedy) [22:56:56] (03PS3) 10Reedy: Simplify config, remove additions/removal upto REL1_23 [tools/release] - 10https://gerrit.wikimedia.org/r/343589 [22:57:16] (03PS2) 10Reedy: After REL1_23 is obsolete, simplify config further [tools/release] - 10https://gerrit.wikimedia.org/r/343591 [23:06:42] (03CR) 10Chad: "Not quite correct, as 1.23 still needs to be technically built until it's EOL'd" [tools/release] - 10https://gerrit.wikimedia.org/r/343591 (owner: 10Reedy) [23:07:22] (03CR) 10Chad: "Oh, duh. I see it." [tools/release] - 10https://gerrit.wikimedia.org/r/343591 (owner: 10Reedy)