[00:00:12] Project beta-update-databases-eqiad build #17269: 04STILL FAILING in 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17269/ [00:16:53] Yippee, build fixed! [00:16:54] Project selenium-Flow » chrome,beta,Linux,BrowserTests build #399: 09FIXED in 53 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/399/ [00:17:08] Yippee, build fixed! [00:17:08] Project selenium-Flow » firefox,beta,Linux,BrowserTests build #399: 09FIXED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/399/ [00:20:23] Project beta-update-databases-eqiad build #17270: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17270/ [01:20:23] Project beta-update-databases-eqiad build #17271: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17271/ [01:27:35] Commonswiki is upset with Flow ^ [01:27:54] Dropping some keys/indexes that don't exist [01:28:01] Should probably include IF EXISTS in patch files [01:47:25] 06Release-Engineering-Team (Watching / External), 10Architecture, 06Developer-Relations, 10MediaWiki-API, and 2 others: Standardise procedures for deprecating public-facing code - https://phabricator.wikimedia.org/T114384#3285343 (10Anomie) > Like the newly formed MW Platform team for MW code deprecations.... [02:02:38] 10Gerrit, 10MediaWiki-extensions-Other, 06Repository-Admins, 07Technical-Debt: Archive PageLanguageApi extension - https://phabricator.wikimedia.org/T160371#3285357 (10SamanthaNguyen) Looks like June is coming up soon..I'll set up a reminder to work on this when 1.29 comes around! [02:20:23] Project beta-update-databases-eqiad build #17272: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17272/ [03:20:21] Project beta-update-databases-eqiad build #17273: 04STILL FAILING in 20 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17273/ [04:07:38] Yippee, build fixed! [04:07:39] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #400: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/400/ [04:20:13] Project beta-update-databases-eqiad build #17274: 04STILL FAILING in 13 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17274/ [04:42:04] 10MediaWiki-Codesniffer: MediaWiki.NamingConventions.LowerCamelFunctionsName.FunctionName is incorrectly saying "pg_array_parse" should use lower camel - https://phabricator.wikimedia.org/T164533#3285371 (10Legoktm) 05Open>03declined The sniff is correct. That function is not a PHP built-in, it's just a priv... [04:46:27] 10MediaWiki-Codesniffer, 06Community-Tech, 13Patch-For-Review: Undefined index: scope_opener in MediaWiki/Sniffs/Usage/ExtendClassUsageSniff.php - https://phabricator.wikimedia.org/T154731#2922176 (10Legoktm) @brion what version of mediawiki-codesniffer is in your composer.lock? If it's not 0.8.0 you need to... [04:53:43] (03PS1) 10Legoktm: Disallow "`and` and `or` [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355182 (https://phabricator.wikimedia.org/T143888) [04:54:05] (03PS2) 10Legoktm: Disallow `and` and `or` [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355182 (https://phabricator.wikimedia.org/T143888) [04:54:33] 10MediaWiki-Codesniffer, 13Patch-For-Review: Detect "and" and "or" tokens used in PHP code - https://phabricator.wikimedia.org/T143888#2582185 (10Legoktm) a:03Legoktm [05:20:21] Project beta-update-databases-eqiad build #17275: 04STILL FAILING in 21 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17275/ [05:22:07] (03PS1) 10Legoktm: Add sniff to enforce "function (" for closures [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355183 (https://phabricator.wikimedia.org/T149623) [05:22:24] 10MediaWiki-Codesniffer, 13Patch-For-Review: No sniff for "function (" versus "function(" - https://phabricator.wikimedia.org/T149623#2758221 (10Legoktm) a:03Legoktm [06:20:23] Project beta-update-databases-eqiad build #17276: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17276/ [06:21:20] Project selenium-Wikibase » chrome,test,Linux,BrowserTests build #369: 04FAILURE in 1 hr 41 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=test,PLATFORM=Linux,label=BrowserTests/369/ [06:44:58] Project selenium-Wikibase » chrome,beta,Linux,BrowserTests build #369: 04FAILURE in 2 hr 4 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/369/ [06:56:09] 06Release-Engineering-Team (Kanban), 10MediaWiki-extensions-TocTree, 05MW-1.27-release (WMF-deploy-2016-01-12_(1.27.0-wmf.10)): TocTree should pass jshint - https://phabricator.wikimedia.org/T63640#3285434 (10Fomafix) [07:00:17] 10Beta-Cluster-Infrastructure: Steward Rights on Beta Cluster for AlvaroMolina - https://phabricator.wikimedia.org/T165917#3285437 (10AlvaroMolina) a:05Krenair>03None [07:11:33] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:20:26] Project beta-update-databases-eqiad build #17277: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17277/ [07:40:40] godog: outcome of the testing with alerts-beta-wm ? what should I do with shinken-wm (yet)? [07:41:38] Seems to be a flow change breaking db update [07:42:28] ofc, no obvious recent commits [07:42:39] Reedy: https://gerrit.wikimedia.org/r/#/c/320639/ maaaaybe? [07:42:55] Nah, it's trying to drop [07:43:08] A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? \nQuery: ALTER TABLE `flow_ext_ref` DROP KEY flow_ext_ref_idx_v2\n\nFunction: Wikimedia\\Rdbms\\Database::sourceFile( /srv/mediawiki-staging/php-master/extensions/Flow/db_patches/patch-ref_target_not_null.sql )\nError: 1091 Can't DROP 'flow_ext_ref_idx_v2'; check that column/key exists (10.68.23.30) [07:43:18] Something in core that's an unexpected breaking change perhaps. [07:43:39] https://github.com/wikimedia/mediawiki-extensions-Flow/commit/5bf7057a1a01967aa05f6296cf850587ffedb419 [07:43:44] Last change was a while ago to that patch [07:44:13] $updater->modifyExtensionField( 'flow_ext_ref', 'ref_target', "$dir/db_patches/patch-ref_target_not_null.sql" ); [07:44:23] Did something change in Beta Cluster that triggered that path for the first time? [07:45:21] 06Release-Engineering-Team (Watching / External), 10Architecture, 06Developer-Relations, 10MediaWiki-API, and 2 others: Standardise procedures for deprecating public-facing code - https://phabricator.wikimedia.org/T114384#3285512 (10greg) Yeah. My initial reaction was "find the right team to be responsible... [08:00:59] (03PS1) 10Hashar: Remove HHVM from Trusty images [integration/config] - 10https://gerrit.wikimedia.org/r/355187 [08:14:46] PROBLEM - DPKG on contint1001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [08:19:43] !log Regenerated Nodepool base image for Trusty. Got rid of hhvm from it [08:19:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:20:09] !log Updating Nodepool snapshot-ci-trusty [08:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:20:26] Project beta-update-databases-eqiad build #17278: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17278/ [08:24:26] PROBLEM - puppet last run on contint1001 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[initramfs-tools],Package[openjdk-7-jdk] [08:25:45] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Kanban), 13Patch-For-Review: Puppet broken on CI Trusty nodes: Could not find dependency Apt::Pin[hhvm-from-experimental] - https://phabricator.wikimedia.org/T165462#3285562 (10hashar) I have removed HHVM from the Trusty imag entirely with... [08:25:47] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Allow use of ext-gmp on CI for composer tests - https://phabricator.wikimedia.org/T164977#3285565 (10hashar) [08:25:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Kanban), 13Patch-For-Review: Puppet broken on CI Trusty nodes: Could not find dependency Apt::Pin[hhvm-from-experimental] - https://phabricator.wikimedia.org/T165462#3285563 (10hashar) 05Open>03Resolved a:03hashar [08:29:46] RECOVERY - DPKG on contint1001 is OK: All packages OK [08:32:16] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Kanban), 13Patch-For-Review: Allow use of ext-gmp on CI for composer tests - https://phabricator.wikimedia.org/T164977#3285598 (10hashar) 05Open>03Resolved a:03hashar Should be good now :) [08:37:33] (03PS3) 10Hashar: Add PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355124 (https://phabricator.wikimedia.org/T104309) (owner: 10Ladsgroup) [08:39:23] (03CR) 10Hashar: [C: 032] Add PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355124 (https://phabricator.wikimedia.org/T104309) (owner: 10Ladsgroup) [08:40:25] (03Merged) 10jenkins-bot: Add PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355124 (https://phabricator.wikimedia.org/T104309) (owner: 10Ladsgroup) [08:44:01] greg-g: Hi! do you have a minute for a qq? [08:44:27] (I saw you online earlier on, not sure if you are in eu timezone now or not) [08:45:52] 06Release-Engineering-Team (Kanban), 06Performance-Team, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3285627 (10Gilles) [08:46:07] 06Release-Engineering-Team (Kanban), 06Performance-Team, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3285641 (10Gilles) p:05Triage>03Normal [08:46:45] (03PS1) 10Hashar: mw-tools-codesniffer-mwcore-testrun: use HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/355189 (https://phabricator.wikimedia.org/T157750) [08:48:20] (03PS2) 10Hashar: mw-tools-codesniffer-mwcore-testrun: use HHVM [integration/config] - 10https://gerrit.wikimedia.org/r/355189 (https://phabricator.wikimedia.org/T157750) [08:49:06] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3285646 (10Peachey88) [08:50:19] (03CR) 10Hashar: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [08:53:26] RECOVERY - puppet last run on contint1001 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [08:53:46] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 19 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [08:54:20] 06Release-Engineering-Team (Kanban), 06Performance-Team: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3285669 (10Gilles) [08:54:48] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3285683 (10Gilles) a:05greg>03mmodell [08:55:01] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator: Enable storage and display of OGV and WEBM videos on Phabricator - https://phabricator.wikimedia.org/T166112#3285686 (10Gilles) [08:56:01] (03CR) 10jerkins-bot: [V: 04-1] Update for CodeSniffer 3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [08:58:11] 06Release-Engineering-Team (Kanban), 06Performance-Team, 10Phabricator, 15User-greg: Create Performance phame blog - https://phabricator.wikimedia.org/T166110#3285627 (10Peachey88) Also need to re-add the blog widget back default front page: {T163067} [09:04:52] (03CR) 10Hashar: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [09:05:46] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [09:15:15] (03CR) 10jerkins-bot: [V: 04-1] Update for CodeSniffer 3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [09:15:42] Project beta-scap-eqiad build #156504: 15ABORTED in 1 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156504/ [09:20:31] Project beta-update-databases-eqiad build #17279: 04STILL FAILING in 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17279/ [09:23:43] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [09:23:57] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:23:59] Project beta-scap-eqiad build #156505: 04FAILURE in 0.64 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156505/ [09:33:55] Project beta-scap-eqiad build #156506: 04STILL FAILING in 0.58 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156506/ [10:00:48] Yippee, build fixed! [10:00:49] Project beta-scap-eqiad build #156507: 09FIXED in 20 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/156507/ [10:03:43] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [10:03:58] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:05:47] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:18:46] (03CR) 10WMDE-leszek: Add extension-phan-generic to Wikibase CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/341304 (owner: 10Addshore) [10:20:33] Project beta-update-databases-eqiad build #17280: 04STILL FAILING in 33 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17280/ [10:40:48] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [10:50:20] (03PS2) 10Addshore: Add extension-phan-generic to Wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/341304 [10:52:29] (03CR) 10WMDE-leszek: [C: 031] Add extension-phan-generic to Wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/341304 (owner: 10Addshore) [11:20:28] Project beta-update-databases-eqiad build #17281: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17281/ [11:34:15] ores needs lots of love in beta [11:34:21] it's in my todo list [11:37:17] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1970 bytes in 0.060 second response time [11:41:37] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3285943 (10Nemo_bis) > But maybe seeing the messages appearing in the Incubator UI somehow... [11:42:21] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 33349 bytes in 4.859 second response time [11:43:49] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:20:24] Project beta-update-databases-eqiad build #17282: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17282/ [12:20:48] (03CR) 10Krinkle: "@Peter: Where is this used?" [integration/config] - 10https://gerrit.wikimedia.org/r/354385 (owner: 10Phedenskog) [12:21:54] (03CR) 10Phedenskog: "It's not used yet (only when I'm testing it locally), wanted to make sure this time that everything is on Jenkins before we push :)" [integration/config] - 10https://gerrit.wikimedia.org/r/354385 (owner: 10Phedenskog) [12:22:18] (03CR) 10Phedenskog: [C: 04-1] "Wait with this a while since I've seen the current instance that we wanna use to test is too small." [integration/config] - 10https://gerrit.wikimedia.org/r/354385 (owner: 10Phedenskog) [12:23:01] (03CR) 10Hashar: [C: 032] "Das ist gut !!!" [integration/config] - 10https://gerrit.wikimedia.org/r/341304 (owner: 10Addshore) [12:23:25] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3286036 (10Amqui) Nemo_bis I'm not saying those things are useless and not important. I'm... [12:23:49] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [12:24:30] (03Merged) 10jenkins-bot: Add extension-phan-generic to Wikibase CI [integration/config] - 10https://gerrit.wikimedia.org/r/341304 (owner: 10Addshore) [12:24:57] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:24:58] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:29:35] (03PS1) 10Hashar: Switch mediawiki-core-phpcs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/355207 [12:32:54] Is there docs on alerts-beta-wm ?? I cant seem to find any and im kinda curious bout that bot [12:36:01] (03PS2) 10Hashar: Switch mediawiki-core-phpcs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/355207 [12:45:21] (03PS3) 10Hashar: Switch mediawiki-core-phpcs to Jessie [integration/config] - 10https://gerrit.wikimedia.org/r/355207 [12:46:54] (03PS1) 10Hashar: Fix mediawiki-core-phpcs-jessie node assignement [integration/config] - 10https://gerrit.wikimedia.org/r/355211 [12:49:55] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3286079 (10Nikerabbit) >>! In T165585#3286036, @Amqui wrote: > For example, including Tran... [12:54:56] (03CR) 10Dereckson: "This change has the positive side effect to fix the PHPUnit tests (launched through `composer test`) on my install: they don't pass all b" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [12:55:05] (03CR) 10Dereckson: [C: 031] Disallow `and` and `or` [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355182 (https://phabricator.wikimedia.org/T143888) (owner: 10Legoktm) [12:59:56] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:00:59] (03PS1) 10Hashar: Fix Gerrit reported message for codesniffer testrun [integration/config] - 10https://gerrit.wikimedia.org/r/355215 [13:11:21] 06Release-Engineering-Team (Watching / External), 10Architecture, 06Developer-Relations, 10MediaWiki-API, and 2 others: Standardise procedures for deprecating public-facing code - https://phabricator.wikimedia.org/T114384#3286117 (10Qgil) Alright, I agree it is complex. This is the main scenario that I thi... [13:16:04] Zppix: no docs :( it is an experiment on prometheus alerting I ran for the hackathon [13:16:23] godog: whats its based on or how does it work [13:20:28] Project beta-update-databases-eqiad build #17283: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17283/ [13:22:05] (03CR) 10Hashar: [C: 032] "Ideally the test would also validate that || and && are valid :] Well done on that one, I am surprised it hasn't been enabled previously." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355182 (https://phabricator.wikimedia.org/T143888) (owner: 10Legoktm) [13:24:05] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:24:30] Zppix: based on this https://github.com/prometheus/alertmanager https://gerrit.wikimedia.org/r/#/c/354460/ and the former sends webhooks via http to a custom irc bot [13:25:55] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:28:21] (03CR) 10Hashar: [C: 032] Fix Gerrit reported message for codesniffer testrun [integration/config] - 10https://gerrit.wikimedia.org/r/355215 (owner: 10Hashar) [13:30:17] (03Merged) 10jenkins-bot: Fix Gerrit reported message for codesniffer testrun [integration/config] - 10https://gerrit.wikimedia.org/r/355215 (owner: 10Hashar) [13:34:59] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [13:47:07] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3270613 (10Nemo_bis) [13:49:55] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3286266 (10Nemo_bis) > Nemo_bis I'm not saying those things are useless and not important.... [14:00:56] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:06:15] (03Abandoned) 10Hashar: Revert "Rely on `php` in assert-phpflavor macro" [integration/config] - 10https://gerrit.wikimedia.org/r/337225 (https://phabricator.wikimedia.org/T157750) (owner: 10Paladox) [14:07:01] (03Restored) 10Hashar: Whitelist Alexia [integration/config] - 10https://gerrit.wikimedia.org/r/332002 (owner: 10Paladox) [14:07:50] (03PS4) 10Hashar: Whitelist Alexia [integration/config] - 10https://gerrit.wikimedia.org/r/332002 (owner: 10Paladox) [14:07:52] (03PS4) 10Hashar: Whitelist tosfos [integration/config] - 10https://gerrit.wikimedia.org/r/343403 (owner: 10Paladox) [14:07:59] (03CR) 10Hashar: [C: 032] Whitelist tosfos [integration/config] - 10https://gerrit.wikimedia.org/r/343403 (owner: 10Paladox) [14:08:02] (03CR) 10Hashar: [C: 032] Whitelist Alexia [integration/config] - 10https://gerrit.wikimedia.org/r/332002 (owner: 10Paladox) [14:10:14] (03Merged) 10jenkins-bot: Whitelist Alexia [integration/config] - 10https://gerrit.wikimedia.org/r/332002 (owner: 10Paladox) [14:10:36] (03Merged) 10jenkins-bot: Whitelist tosfos [integration/config] - 10https://gerrit.wikimedia.org/r/343403 (owner: 10Paladox) [14:11:51] (03Abandoned) 10Hashar: Add new mediawiki-core-composer-test-HEAD-php55lint-trusty test [integration/config] - 10https://gerrit.wikimedia.org/r/339666 (https://phabricator.wikimedia.org/T158974) (owner: 10Paladox) [14:14:46] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:20:23] Project beta-update-databases-eqiad build #17284: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17284/ [14:29:35] PROBLEM - Host deployment-phab02 is DOWN: CRITICAL - Host Unreachable (10.68.19.232) [14:33:45] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #401: 04FAILURE in 1 min 45 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/401/ [14:34:06] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:34:26] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:36:34] thcipriani|afk, i've +2ed https://gerrit.wikimedia.org/r/#/c/333997/ can you wait to cut the release branch till that merges? thanks. [14:37:17] subbu: merge failed [14:37:27] ugh .. ok. let me look. [14:37:28] Needs rebasing [14:37:33] subbu: yup, np, I generally cut the branch at 17 UTC [14:38:16] ok .. i have some time then to eat breakfast and get back to this then. :) [14:39:54] Almost certainly it's a release-notes thing [14:41:07] hashar thanks [14:43:57] subbu: Yeah, it was [14:44:55] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:54:52] PROBLEM - Puppet errors on deployment-eventlogging03 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:01:01] Reedy, ty :) [15:04:50] RECOVERY - Puppet errors on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:09:28] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:12:30] I've moved alerts-beta-wm to ##wikimedia-alertmanager btw to avoid even more alert spam fatigue [15:14:32] Yippee, build fixed! [15:14:32] Project selenium-CentralNotice » chrome,beta,Linux,BrowserTests build #403: 09FIXED in 30 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/403/ [15:14:41] Yippee, build fixed! [15:14:42] Project selenium-CentralNotice » firefox,beta,Linux,BrowserTests build #403: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/403/ [15:15:16] Yippee, build fixed! [15:15:16] Project selenium-CentralNotice » chrome,beta,Windows 7,BrowserTests build #403: 09FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=BrowserTests/403/ [15:15:17] Yippee, build fixed! [15:15:17] Project selenium-CentralNotice » chrome,beta,OS X 10.9,BrowserTests build #403: 09FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/403/ [15:15:23] yum [15:16:09] Yippee, build fixed! [15:16:09] Project selenium-CentralNotice » firefox,beta,Windows 7,BrowserTests build #403: 09FIXED in 2 min 7 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Windows%207,label=BrowserTests/403/ [15:16:24] Yippee, build fixed! [15:16:24] Project selenium-CentralNotice » firefox,beta,OS X 10.9,BrowserTests build #403: 09FIXED in 2 min 22 sec: https://integration.wikimedia.org/ci/job/selenium-CentralNotice/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/403/ [15:19:48] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:20:26] Project beta-update-databases-eqiad build #17285: 04STILL FAILING in 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17285/ [15:24:56] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [15:25:45] A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? \nQuery: ALTER TABLE `flow_ext_ref` DROP KEY flow_ext_ref_idx_v2\n\nFunction: Wikimedia\\Rdbms\\Database::sourceFile( /srv/mediawiki-staging/php- [15:27:49] (03PS1) 10Hashar: .dockerignore for git/local files [integration/quibble] - 10https://gerrit.wikimedia.org/r/355235 [15:28:43] (03PS1) 10Hashar: Strip out temp /opt/quibble from image [integration/quibble] - 10https://gerrit.wikimedia.org/r/355236 [15:34:23] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:45:50] 06Release-Engineering-Team, 07Jenkins: mediawiki-core-phpcs-trusty broken, can't find hhvm in path - https://phabricator.wikimedia.org/T166144#3286545 (10Gilles) [15:46:04] 06Release-Engineering-Team, 07Jenkins: mediawiki-core-phpcs-trusty broken, can't find hhvm in path - https://phabricator.wikimedia.org/T166144#3286557 (10Gilles) p:05Triage>03Unbreak! [15:46:46] 06Release-Engineering-Team (Kanban), 07Jenkins: mediawiki-core-phpcs-trusty broken, can't find hhvm in path - https://phabricator.wikimedia.org/T166144#3286560 (10hashar) a:03hashar I thought I had fixed it :( [15:46:47] gilles: looking at it [15:48:43] 06Release-Engineering-Team (Kanban), 07Jenkins: mediawiki-core-phpcs-trusty broken, can't find hhvm in path - https://phabricator.wikimedia.org/T166144#3286566 (10hashar) 05Open>03Resolved I have migrated that job from trusty to jessie. It uses HHVM and the version on Trusty is obsolete / got removed 200... [15:49:01] gilles: il faut juste faire un recheck. J'ai tout cassé ce matin :( [15:49:43] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3286570 (10Verdy_p) In fact many languages are now supported only in MediaWiki, when minor... [15:52:22] 06Release-Engineering-Team (Kanban), 07Jenkins: mediawiki-core-phpcs-trusty broken, can't find hhvm in path - https://phabricator.wikimedia.org/T166144#3286572 (10hashar) Gilles and I have recheck the couple jobs that still had a verified -1 due to the issue: https://gerrit.wikimedia.org/r/#/c/354274/ https:/... [15:59:11] (03PS1) 10Hashar: Reorder commands in Dockerfile [integration/quibble] - 10https://gerrit.wikimedia.org/r/355241 [15:59:17] thcipriani: ^^ :] [15:59:26] it is really all very horrible [15:59:30] :) [15:59:33] oh good [16:03:35] 06Release-Engineering-Team, 10Quibble: Start mysql in quibble container - https://phabricator.wikimedia.org/T166145#3286617 (10hashar) [16:09:22] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:26] Project beta-update-databases-eqiad build #17286: 04STILL FAILING in 26 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17286/ [16:31:41] (03CR) 10Thcipriani: [C: 031] "Made some comments inline." (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/355241 (owner: 10Hashar) [16:35:29] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:36:05] 06Release-Engineering-Team (Kanban), 05Deployment Blockers, 05Release: MW-1.30.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T163512#3286710 (10thcipriani) [16:51:08] (03CR) 10Legoktm: "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [16:52:39] what is quibble? [16:53:19] (03CR) 10Thcipriani: [C: 032] .dockerignore for git/local files [integration/quibble] - 10https://gerrit.wikimedia.org/r/355235 (owner: 10Hashar) [16:53:58] legoktm: hackathon project, pretty much the idea (as far as I understand it :P) is to slowly deprecate integration/jenkins and move it into python [16:54:40] the giant mess of bash scripts and stuff? [16:54:47] yeah [16:54:52] (03Merged) 10jenkins-bot: .dockerignore for git/local files [integration/quibble] - 10https://gerrit.wikimedia.org/r/355235 (owner: 10Hashar) [16:55:36] !log dropped flow_ext_ref from commonswiki on beta. schema migration is busted, going to let it recreate table [16:55:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:55:48] so rather than have a bunch of bash scripts both inline and in slave-scripts that are called via JJB, just call quibble [16:55:50] !log there was no data [16:55:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:56:44] Project beta-update-databases-eqiad build #17287: 04STILL FAILING in 18 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17287/ [16:56:46] quibble, currently, also has zuul-cloner as a dependency, so the possibility to move other responsibility from zuul/jenkins into quibble is a thing [16:59:01] Project beta-update-databases-eqiad build #17288: 04STILL FAILING in 3.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17288/ [16:59:32] Project beta-update-databases-eqiad build #17289: 04STILL FAILING in 2.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17289/ [16:59:54] replag? [16:59:54] wtf. [17:00:25] Project beta-update-databases-eqiad build #17290: 04STILL FAILING in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17290/ [17:00:34] wtf did I do? [17:01:12] Project beta-update-databases-eqiad build #17291: 04STILL FAILING in 2.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17291/ [17:10:27] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:10:51] Grrrrrr [17:10:58] I broke the slave somehow [17:20:06] Project beta-update-databases-eqiad build #17292: 04STILL FAILING in 5.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17292/ [17:21:18] (03CR) 10jerkins-bot: [V: 04-1] Update for CodeSniffer 3.0 [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [17:24:43] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [17:26:57] RECOVERY - Puppet staleness on deployment-prometheus01 is OK: OK: Less than 1.00% above the threshold [3600.0] [17:36:52] hi folks! [17:37:10] we seem to have some connection issues with packagist today [17:37:14] The "https://packagist.org/p/provider-2017-04%2486c7cf8d14faebc894d0a52237f6873b18b913101a4d6b8f50fbac618900cd15.json" file could not be downloaded: Failed to enable crypto [17:37:24] https://integration.wikimedia.org/ci/job/wikimedia-fundraising-crm-composer-php55-trusty/97/console [17:37:50] enable crypto? :p [17:37:52] jk, hmmmm [17:38:31] in my day we downloaded packages over http! [17:39:26] So quick google makes it sound like bad proxying [17:43:35] (03PS1) 10Ladsgroup: Remove browser tests from PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355254 [17:45:05] RainbowSprinkles: would we want to ask about that in -operations? [17:45:14] hasharAway: for when you're back, please check out https://gerrit.wikimedia.org/r/#/c/355254/ super straightforward and it's blocking me as jenkins is broken on master :D [17:45:56] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [17:49:16] ejegg: Nah, it's probably on our end :) [17:49:43] Is it just that one job? Others look to be passing [17:49:47] Could've been transient? [17:59:41] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [18:20:03] Project beta-update-databases-eqiad build #17293: 04STILL FAILING in 2.8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17293/ [18:20:54] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [18:28:17] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2135 bytes in 0.420 second response time [18:29:17] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1571 bytes in 0.371 second response time [18:29:23] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2135 bytes in 0.417 second response time [18:31:17] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1572 bytes in 1.294 second response time [18:31:21] PROBLEM - App Server Main HTTP Response on deployment-mediawiki05 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1572 bytes in 1.395 second response time [18:31:44] Hmm ^^ [18:32:05] RainbowSprinkles ^^ [18:32:17] I saw. [18:32:21] ok [18:33:32] RainbowSprinkles: looks like the problem is pretty consistent on the trusty CI boxes: https://integration.wikimedia.org/ci/job/mwext-donationinterfacecore-REL1_27-testextension-zend55/ [18:33:53] rel1_27, bleh [18:34:03] php55 [18:34:07] much puke [18:34:09] so sad [18:35:49] well, it's LTS! [18:36:40] and.... one of those finally worked [18:36:53] so, who knows... [18:39:29] LTS, heh. [18:57:23] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3287178 (10Aklapper) @Verdy_p: Please structure longer comments (e.g. by using [[ https://... [19:05:24] (03CR) 10Legoktm: "CI failure is bogus" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [19:20:04] Project beta-update-databases-eqiad build #17294: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17294/ [19:42:58] 06Release-Engineering-Team, 06Language-Team, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-WikimediaIncubator, and 2 others: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585#3287297 (10Verdy_p) It is structured, each paragraph has its topic. [19:47:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<20.00%) [19:48:00] (03CR) 10Paladox: [C: 031] "recheck" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355067 (https://phabricator.wikimedia.org/T142474) (owner: 10Legoktm) [19:51:15] RainbowSprinkles: I wonder if it's reasonable to increase Nodepool quota somewhat. At least more than 25. It seems unrealistic that we should always have <25 concurrent jobs running. Especially now that all jobs have been migrated from legacy slaves to Nodepool. [19:51:22] This number was first picked when 50% of jobs was still on CI slaves. [19:51:51] I wonder if there is a task for this already or what it would be blocked on. [19:52:49] We used to have 7 or 8 permanent medium-size slaves with 4 slots, which is like 40 slots total. Now we have 25 small slaves with 1 executor each. [19:53:08] I have no control over this. [19:53:16] Quota is up to cloud team, not us [19:56:58] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:03:51] RainbowSprinkles: Looks like our current quota is 29, not 25. [20:04:09] But I imagine that also we have not yet asked or decided, the request needs to come from RelEng. [20:04:24] At least within the current quota it seems that Nodepool config could be raised a bit. [20:08:01] Krinkle: IIRC there was something about several instances being stuck in delete that may explain the mismatch between the current quota and the nodepool config. Nodepool puts quite a bit of stress on openstack infra so ramping up quota is something that would speed up CI, but cloud team has been understandably wary about. [20:09:08] the "stress" for openstack, I guess mainly relates to nodepool churn, but this is me conveying what's been conveyed to me since I don't have a lot of 1st-hand knowledge of the pressure points of openstack [20:09:53] I think the easiest wins for CI would be amalgamating jobs that have the same setup where possible [20:10:53] in that way the overhead of nodepool instance setup at the beginning of the test isn't incurred again and again for very similar jobs where possible [20:19:17] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 46499 bytes in 0.924 second response time [20:19:21] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47102 bytes in 0.822 second response time [20:21:03] Project beta-update-databases-eqiad build #17295: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17295/ [20:21:08] Replication is super lagged in beta, but should start fixing soon [20:21:18] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 46525 bytes in 2.305 second response time [20:21:20] RECOVERY - App Server Main HTTP Response on deployment-mediawiki05 is OK: HTTP OK: HTTP/1.1 200 OK - 46525 bytes in 3.171 second response time [20:21:32] That'll still fail for a bit though until replag goes down [20:22:02] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:22:57] thcipriani: Yeah, there've been definitely a number of initiatives (and more on the way in phab) about simpifying jobs and speeding up jobs. [20:23:26] But even then I think a concurrency of 25 is quite small for our commit activity. Something within the same order of magnitude (e.g. 30 or 50) would help a lot. [20:23:59] this is true [20:31:57] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:03] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:41:41] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #403: 04FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/403/ [20:41:45] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #403: 04FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/403/ [20:55:07] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:21:07] Project beta-update-databases-eqiad build #17296: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17296/ [21:30:08] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:49] (03CR) 10Hashar: [C: 032] Remove browser tests from PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355254 (owner: 10Ladsgroup) [22:04:46] (03Merged) 10jenkins-bot: Remove browser tests from PropertySuggester [integration/config] - 10https://gerrit.wikimedia.org/r/355254 (owner: 10Ladsgroup) [22:16:16] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1592 bytes in 0.098 second response time [22:16:19] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2156 bytes in 0.099 second response time [22:17:45] Oh shut it shinken [22:17:57] lol [22:21:03] Project beta-update-databases-eqiad build #17297: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17297/ [22:21:18] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 46573 bytes in 1.000 second response time [22:21:22] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47136 bytes in 0.912 second response time [22:22:56] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [22:31:54] 10MediaWiki-Codesniffer: Closure formatting is ugly - https://phabricator.wikimedia.org/T154789#3287796 (10MtDu) Have no idea how to approach this, but am willing it to take it on. @Legoktm Any guidance you can give me on how to tackle this? [22:39:18] Ok, I'm stumped. There's a stupid query that seems to be the source of the replag. It keeps attempting (and failing?) [22:39:22] I think it came from update.php [22:39:52] USE enwikisource; UPDATE /* Wikimedia\Rdbms\Database::query */ page SET page_content_model = 'proofread-index' WHERE page_namespace = 106 AND page_content_model = 'wikitext' ORDER BY page_namespace, page_title LIMIT 1000; [22:40:27] (why would we be mass-updating page_content_models outside of updates?) [22:40:34] Failing...because of the LIMIT on UPDATE? [22:40:41] Or because 0 rows updated? [22:51:49] (03PS1) 10MtDu: Allow one line closures for functions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/355368 (https://phabricator.wikimedia.org/T154789) [23:02:56] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:21:02] Project beta-update-databases-eqiad build #17298: 04STILL FAILING in 1 min 2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/17298/ [23:46:30] greg-g: Hi! I tentatively reserved a one-time CentralNotice deploy slot for this Thursday. It's for a change that's too big for a SWAT, but that shouldn't really go on the train because it's best deployed to all wikis at once... Just sending e-mail now (as per request on Deployments page...)