[00:45:12] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:20:02] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [01:49:48] RECOVERY - Puppet run on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [02:19:13] Project selenium-QuickSurveys ยป chrome,beta,Linux,,contintLabsSlave && UbuntuTrusty build #21: 04FAILURE in 6 min 13 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,VERSION=,label=contintLabsSlave%20&&%20UbuntuTrusty/21/ [03:13:39] Yippee, build fixed! [03:13:40] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #939: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/939/ [04:31:43] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #781: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/781/ [06:23:34] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2194207 (10Joe) p:5Triage>3Low [06:23:44] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2192922 (10Joe) a:5Joe>3None [06:24:14] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2192922 (10Joe) @Krenair please stop assigning me tickets instead of subscribing me to those [06:24:34] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2194211 (10Joe) p:5Triage>3Lowest a:5Joe>3None [06:24:52] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2192909 (10Joe) Again, stop assigning vs subscribing people to tickets. [06:58:32] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:15:57] 10Beta-Cluster-Infrastructure, 6Operations, 7Performance: Need a way to simulate replication lag to test replag issues - https://phabricator.wikimedia.org/T40945#2194273 (10Nikerabbit) [07:33:03] RECOVERY - Puppet run on deployment-poolcounter01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:33] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [07:42:40] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [08:11:11] 10Continuous-Integration-Config, 10BlueSpice: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#2194298 (10Peachey88) [11:15:13] 11:10:27 IOError: Lock at '/mnt/jenkins-workspace/workspace/mediawiki-extensions-qunit/src/extensions/cldr/.git/refs/heads/wmf/1.24wmf1.lock' could not be obtained [11:15:16] 11:10:27 Build step 'Execute shell' marked build as failure [11:15:26] https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/38364/console [12:44:13] PROBLEM - Host integration-dev is DOWN: CRITICAL - Host Unreachable (10.68.17.81) [13:21:25] Project beta-scap-eqiad build #97751: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97751/ [13:26:08] Project beta-scap-eqiad build #97752: 04STILL FAILING in 2 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97752/ [13:30:54] Project beta-scap-eqiad build #97753: 04STILL FAILING in 2 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97753/ [13:30:58] 10Browser-Tests-Infrastructure, 6Release-Engineering-Team, 7Epic, 13Patch-For-Review, 7Tracking: Fix scenarios that fail at en.wikipedia.beta.wmflabs.org or do not run them daily - https://phabricator.wikimedia.org/T94150#2018106 (10zeljkofilipin) a:5zeljkofilipin>3None [13:37:31] Project beta-scap-eqiad build #97754: 04STILL FAILING in 2 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97754/ [13:46:41] I have disabled daily builds, e-mail and irc notifications for all deployed selenium* jobs [13:46:42] https://integration.wikimedia.org/ci/view/selenium/ [13:46:47] you are welcome ;) [13:47:08] and apologies for keeping forgetting to do that :| [13:47:37] Project beta-scap-eqiad build #97755: 04STILL FAILING in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97755/ [13:52:51] 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 3Wikidata-Sprint-2016-03-01, 3Wikidata-Sprint-2016-04-12: [Task] Move Wikidata browsertests into Wikibase repository - https://phabricator.wikimedia.org/T118727#2194830 (10Tobi_WMDE_SW) [13:52:53] 10Browser-Tests-Infrastructure, 13Patch-For-Review: Simplify creating Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2194831 (10zeljkofilipin) [13:52:57] 7Browser-Tests, 10Continuous-Integration-Config, 10Wikidata, 7Story, and 2 others: [Story] Run browsertests regularly on test.wikidata.org via Jenkins - https://phabricator.wikimedia.org/T101497#2194832 (10Tobi_WMDE_SW) [13:57:38] Project beta-scap-eqiad build #97756: 04STILL FAILING in 2 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97756/ [14:01:57] RECOVERY - Puppet run on deployment-upload is OK: OK: Less than 1.00% above the threshold [0.0] [14:03:33] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [14:04:03] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:05:08] (03PS3) 10Zfilipin: Skip Cucumber features or scenarios tagged with `@skip` tag [selenium] - 10https://gerrit.wikimedia.org/r/282370 (https://phabricator.wikimedia.org/T128190) [14:05:58] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [14:06:42] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:07:42] Project beta-scap-eqiad build #97757: 04STILL FAILING in 2 min 38 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97757/ [14:09:45] 10Beta-Cluster-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki: Beta Cluster threw a 'internal_api_error_DBConnectionError' at me - https://phabricator.wikimedia.org/T129192#2194879 (10Jdforrester-WMF) >>! In T129192#2101951, @AlexMonk-WMF wrote: > I don't think there's anything for VE to do here.... [14:09:50] 10Beta-Cluster-Infrastructure, 10VisualEditor, 10VisualEditor-MediaWiki: Beta Cluster threw a 'internal_api_error_DBConnectionError' at me - https://phabricator.wikimedia.org/T129192#2194880 (10Jdforrester-WMF) a:5Jdforrester-WMF>3None [14:17:37] Project beta-scap-eqiad build #97758: 04STILL FAILING in 2 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97758/ [14:25:31] 10Browser-Tests-Infrastructure, 13Patch-For-Review: Simplify creating Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2194912 (10zeljkofilipin) [14:27:28] Project beta-scap-eqiad build #97759: 04STILL FAILING in 2 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97759/ [14:40:03] Project beta-scap-eqiad build #97760: 04STILL FAILING in 5 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97760/ [14:42:21] blerg. [14:42:35] !log deployment-mediawiki01 disk full :( [14:42:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:43:32] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: Name or service not known [14:45:40] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:47:26] Project beta-scap-eqiad build #97761: 04STILL FAILING in 2 min 29 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97761/ [14:47:56] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:48:24] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:48:24] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:48:32] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 29746 bytes in 1.726 second response time [14:50:47] PROBLEM - Puppet run on deployment-mediawiki03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:52:41] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:54:41] PROBLEM - Parsoid on deployment-parsoid05 is CRITICAL: Connection refused [14:55:32] zeljkof: https://phabricator.wikimedia.org/project/view/1938/ [14:56:19] Luke081515: thanks! [14:56:35] zeljkof: Need a herald rule too? [14:56:47] Luke081515: not sure [14:56:53] that is it used for? [14:57:15] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:57:22] zeljkof: Some users got one: https://phabricator.wikimedia.org/T132334#2195008 [14:57:28] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Simplify creating Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2195041 (10zeljkofilipin) [14:57:56] Project beta-scap-eqiad build #97762: 04STILL FAILING in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97762/ [14:57:59] Luke081515: I think I am fine for now [14:58:11] ok, ask if you need on e;) [14:58:12] *one [14:58:26] It's a global herald rule, so only a few people can create them [14:58:51] PROBLEM - Puppet run on integration-publisher is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:59:41] RECOVERY - Parsoid on deployment-parsoid05 is OK: HTTP OK: HTTP/1.1 200 OK - 1514 bytes in 0.099 second response time [14:59:43] PROBLEM - Host Generic Beta Cluster is DOWN: check_ping: Invalid hostname/address - en.wikipedia.beta.wmflabs.org [14:59:57] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:00:10] Luke081515: thanks, will do [15:00:14] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:01:18] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:01:18] PROBLEM - Puppet run on deployment-memc02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:02:58] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [15:03:40] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:04:00] PROBLEM - Puppet run on deployment-poolcounter01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:04:32] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:04:36] RECOVERY - Host Generic Beta Cluster is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [15:04:45] (03PS1) 10Zfilipin: Make site-specific Cucumber tag optional [selenium] - 10https://gerrit.wikimedia.org/r/282709 (https://phabricator.wikimedia.org/T128190) [15:04:56] PROBLEM - Puppet run on deployment-salt is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:05:16] Project beta-scap-eqiad build #97763: 04STILL FAILING in 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97763/ [15:07:06] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:08:33] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:08:47] Project beta-scap-eqiad build #97764: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97764/ [15:08:51] PROBLEM - Puppet run on integration-slave-trusty-1011 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:09:32] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Simplify creating Jenkins jobs for running browser tests daily - https://phabricator.wikimedia.org/T128190#2195079 (10zeljkofilipin) [15:09:35] PROBLEM - Puppet run on deployment-sentry2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:12:01] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:12:35] Project beta-scap-eqiad build #97765: 04STILL FAILING in 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97765/ [15:12:57] PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:12:57] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:15:51] PROBLEM - Puppet run on integration-raita is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:17:29] Project beta-scap-eqiad build #97766: 04STILL FAILING in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97766/ [15:19:05] Project browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce build #392: 04FAILURE in 1 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-CentralNotice-en.wikipedia.beta.wmflabs.org-os_x_10.9-chrome-sauce/392/ [15:19:38] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:20:45] Project beta-scap-eqiad build #97767: 04STILL FAILING in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97767/ [15:25:09] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2195136 (10greg) p:5Low>3Normal (not low for Beta Cluster, where this task has it's "home") :) [15:25:12] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2195139 (10greg) p:5Lowest>3Normal (not lowest for Beta Cluster, where this task has it's "home") :) [15:25:23] Project beta-scap-eqiad build #97768: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97768/ [15:25:34] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [15:27:05] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2195144 (10Krenair) @Joe: Didn't your commit linked above cause this error? [15:28:32] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2195147 (10Krenair) @Joe: Again, didn't one of your commits linked above cause this error? [15:29:08] Project beta-scap-eqiad build #97769: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97769/ [15:30:55] RECOVERY - Puppet run on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:31:05] (03PS2) 10Zfilipin: Make site-specific Cucumber tag optional [selenium] - 10https://gerrit.wikimedia.org/r/282709 (https://phabricator.wikimedia.org/T128190) [15:32:49] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:15] Project beta-scap-eqiad build #97770: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97770/ [15:36:15] 5Gerrit-Migration, 10OOjs-UI: Move the OOUI library's development from gerrit to differential - https://phabricator.wikimedia.org/T122834#2195164 (10greg) [15:36:18] 5Gerrit-Migration, 3releng-201516-q4: Phase 1 repository migrations to Differential (goal - end of June 2016) - https://phabricator.wikimedia.org/T130418#2195163 (10greg) [15:36:25] 5Gerrit-Migration, 10OOjs-UI: Move the OOUI library's development from gerrit to differential - https://phabricator.wikimedia.org/T122834#1915010 (10greg) [15:36:28] 5Gerrit-Migration, 5Gitblit-Deprecate, 6Release-Engineering-Team, 10releng-201516-q3, and 4 others: [RfC]: Migrate code review / management to Phabricator from Gerrit - https://phabricator.wikimedia.org/T119908#2195167 (10greg) [15:37:11] 5Gerrit-Migration, 10Differential: git commit hash remains the same from submitting to after merge - https://phabricator.wikimedia.org/T91420#2195173 (10mmodell) It's not clear from this task why this is desirable or even at all relevant. [15:37:36] 5Gerrit-Migration, 10OOjs-UI: Move the OOUI library's development from gerrit to differential - https://phabricator.wikimedia.org/T122834#1915010 (10greg) (Removing the RFC as a blocker as we plan to migrate some batches simultaneously with the RFC.) [15:37:47] PROBLEM - Puppet run on deployment-restbase03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:38:43] PROBLEM - SSH on integration-slave-trusty-1025 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:39:01] RECOVERY - Puppet run on integration-publisher is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:37] 5Gerrit-Migration, 10Differential, 6translatewiki.net, 7I18n: Train L10n-bot to work with repositories hosted in Wikimedia Phabricator (Diffusion) - https://phabricator.wikimedia.org/T92493#2195186 (10greg) [15:39:55] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:39:56] RECOVERY - Puppet run on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0] [15:40:24] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:41:20] RECOVERY - Puppet run on deployment-memc02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:42:35] 10Continuous-Integration-Config: Add Coren (marc@uberbox.org) to CI V+2ers - https://phabricator.wikimedia.org/T132345#2195189 (10Jdforrester-WMF) [15:43:02] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:32] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:43:44] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:00] RECOVERY - Puppet run on deployment-poolcounter01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:19] 5Gerrit-Migration, 3releng-201516-q4: Phase 1 repository migrations to Differential (goal - end of June 2016) - https://phabricator.wikimedia.org/T130418#2195207 (10greg) [15:44:37] 5Gerrit-Migration, 10Differential, 6translatewiki.net, 7I18n: Train L10n-bot to work with repositories hosted in Wikimedia Phabricator (Diffusion) - https://phabricator.wikimedia.org/T92493#1112862 (10greg) Just spoke with James about this in the office this morning. Siebrand did work to make l10n bot to w... [15:44:39] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:44:40] RECOVERY - Puppet run on deployment-sentry2 is OK: OK: Less than 1.00% above the threshold [0.0] [15:45:22] Project beta-scap-eqiad build #97771: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97771/ [15:47:13] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [15:48:53] RECOVERY - Puppet run on integration-slave-trusty-1011 is OK: OK: Less than 1.00% above the threshold [0.0] [15:51:09] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [15:51:50] 10Deployment-Systems, 6Release-Engineering-Team, 10ReleaseTaggerBot, 7Epic: [EPIC] Code Deploy Dashboard - https://phabricator.wikimedia.org/T280#2195240 (10mmodell) [15:52:01] RECOVERY - Puppet run on integration-slave-precise-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [15:52:59] RECOVERY - Puppet run on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:53:25] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [15:55:00] 6Release-Engineering-Team, 6Project-Admins: SWAT Project (Tag) - https://phabricator.wikimedia.org/T99411#2195278 (10mmodell) 5Open>3stalled @luke081515: Probably both together. @greg: yes ideally differential would be involved. [15:55:14] 10Beta-Cluster-Infrastructure, 7Puppet, 7Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2195281 (10Joe) [15:55:17] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2195280 (10Joe) 5Open>3Resolved [15:55:22] Project beta-scap-eqiad build #97772: 04STILL FAILING in 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97772/ [15:55:51] RECOVERY - Puppet run on integration-raita is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:08] Project beta-scap-eqiad build #97773: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97773/ [15:57:10] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2192909 (10Joe) @Krenair yes, that doesn't mean I should be assigned a ticket. It's just not the way it works. [15:57:34] 10Beta-Cluster-Infrastructure, 7Puppet, 7Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#2195289 (10Joe) [15:57:36] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2195288 (10Joe) 5Open>3Resolved [15:58:56] Project beta-scap-eqiad build #97774: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97774/ [16:05:16] Project beta-scap-eqiad build #97775: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97775/ [16:05:59] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-redis0[12], deployment-sentry2 puppet failures due to missing hiera data redis::shards - https://phabricator.wikimedia.org/T132263#2195340 (10Krenair) For the record it looks like this was fixed in https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deploym... [16:06:16] 10Beta-Cluster-Infrastructure, 7Puppet: deployment-memc0[2-4] puppet failure due to missing hiera data mediawiki::redis_servers::codfw - https://phabricator.wikimedia.org/T132262#2195342 (10Krenair) For the record it looks like this was fixed in https://wikitech.wikimedia.org/w/index.php?title=Hiera:Deployment... [16:15:21] Project beta-scap-eqiad build #97776: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97776/ [16:20:01] Project beta-update-databases-eqiad build #7780: 04FAILURE in 0.74 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7780/ [16:25:21] Project beta-scap-eqiad build #97777: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97777/ [16:27:38] 10Deployment-Systems, 10scap, 6Discovery, 10Scap3 (Scap3-Adoption-Phase1): Deploy discovery-analytics with scap3 - https://phabricator.wikimedia.org/T129149#2195480 (10EBernhardson) Documentation (to be updated for scap3) here: https://wikitech.wikimedia.org/wiki/Discovery/Analytics#How_to_deploy [16:34:19] 5Gerrit-Migration, 10Differential, 6translatewiki.net, 7I18n: Train L10n-bot to work with repositories hosted in Wikimedia Phabricator (Diffusion) - https://phabricator.wikimedia.org/T92493#2195518 (10Nikerabbit) Any repo should do. I just need to know whether Differential uses `git push` or something else... [16:35:20] Project beta-scap-eqiad build #97778: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97778/ [16:42:04] 5Gerrit-Migration, 10Differential: git commit hash remains the same from submitting to after merge - https://phabricator.wikimedia.org/T91420#2195553 (10JanZerebecki) Git works by being content addressable. There is no justification given why a review system would need to change the content arbitrarily. That i... [16:45:22] Project beta-scap-eqiad build #97779: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97779/ [16:46:54] 10Continuous-Integration-Infrastructure, 7Upstream, 7Zuul: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#975087 (10Tgr) Seems to be happening fairly often these days. https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/38439/console ``` 1... [16:53:47] 6Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Create milestones in #releng for quarterly goal work? - https://phabricator.wikimedia.org/T131569#2195606 (10zeljkofilipin) [16:55:20] Project beta-scap-eqiad build #97780: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97780/ [17:05:17] Project beta-scap-eqiad build #97781: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97781/ [17:08:14] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#2195704 (10yuvipanda) AAAAAAAAAAA such a long time ago :D So grains are what salt uses for targeting, and 'cluster' is supposed to set what kind of machines they are... [17:08:42] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#2195706 (10yuvipanda) a:5yuvipanda>3None Unfortunately I have no time to work on this atm :( [17:15:12] 5Gerrit-Migration, 10Differential, 6translatewiki.net, 7I18n: Train L10n-bot to work with repositories hosted in Wikimedia Phabricator (Diffusion) - https://phabricator.wikimedia.org/T92493#2195766 (10greg) >>! In T92493#2195518, @Nikerabbit wrote: > Any repo should do. I just need to know whether Differen... [17:15:18] Project beta-scap-eqiad build #97782: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97782/ [17:16:17] 5Gerrit-Migration, 10Differential, 7Documentation, 7Upstream: Document best practices to amend a change written by another contributor - https://phabricator.wikimedia.org/T121751#2195773 (10mmodell) merged upstream: https://secure.phabricator.com/D15468 `arcanist` now allows you to amend a patch submitted... [17:19:17] 5Gerrit-Migration, 10Differential, 7Documentation, 7Upstream: Document best practices to amend a change written by another contributor - https://phabricator.wikimedia.org/T121751#2195804 (10greg) <3 [17:20:01] Project beta-update-databases-eqiad build #7781: 04STILL FAILING in 0.54 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7781/ [17:20:22] 5Gerrit-Migration, 10Differential, 6translatewiki.net, 7I18n: Train L10n-bot to work with repositories hosted in Wikimedia Phabricator (Diffusion) - https://phabricator.wikimedia.org/T92493#2195813 (10Nikerabbit) >>! In T92493#2195766, @greg wrote: >>>! In T92493#2195518, @Nikerabbit wrote: >> Any repo sho... [17:25:20] Project beta-scap-eqiad build #97783: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97783/ [17:35:17] Project beta-scap-eqiad build #97784: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97784/ [17:45:14] Project beta-scap-eqiad build #97785: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97785/ [17:55:21] Project beta-scap-eqiad build #97786: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97786/ [17:58:49] twentyafterfour, would you mind if I assigned a test task to @20after4 for https://phabricator.wikimedia.org/T111214 ? [17:59:24] Krenair: @mmodell is my 'work account' on phabricator [17:59:47] yes, I said @20after4 specifically because it begins with a number [17:59:55] assign anything you want to 20after4 ;) [17:59:57] that's the bug we're dealing with in wikibugs [18:01:07] I see... [18:05:15] Project beta-scap-eqiad build #97787: 04STILL FAILING in 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97787/ [18:09:33] 07Browser-Tests, 10Reading-Web, 03Reading-Web-Sprint-71-m: Run browser tests on beta cluster from desktop domain - https://phabricator.wikimedia.org/T130429#2195969 (10dr0ptp4kt) [18:15:18] Project beta-scap-eqiad build #97788: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97788/ [18:16:01] 10Beta-Cluster-Infrastructure, 07Puppet: Setup puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#2196009 (10Krenair) We chatted about this on IRC and my impression is that the issue is mostly (?) that production puppetmasters (which have exported resources s... [18:16:09] 10Beta-Cluster-Infrastructure: beta-scap-eqiad scap fails with CentralIdLookup::resetCache may only be called during bootstrapping unit tests - https://phabricator.wikimedia.org/T132370#2196011 (10thcipriani) [18:18:46] 10Beta-Cluster-Infrastructure, 07Puppet: Setup puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#2196043 (10yuvipanda) That's the underlying cause, yes - if they used the same setup, they'd have the same config which would include support for exported resour... [18:19:23] 10Beta-Cluster-Infrastructure, 07Puppet: Setup puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#2196047 (10Krenair) [18:20:01] Project beta-update-databases-eqiad build #7782: 04STILL FAILING in 0.59 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7782/ [18:23:21] 05Gerrit-Migration, 10Differential: git commit hash remains the same from submitting to after merge - https://phabricator.wikimedia.org/T91420#1083174 (10valhallasw) > That in turn moves more trust from the ends to a central location. The way git is used in wikimedia space always moves trust to a central locat... [18:25:19] Project beta-scap-eqiad build #97789: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97789/ [18:35:23] Project beta-scap-eqiad build #97790: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97790/ [18:37:53] 06Release-Engineering-Team, 10MobileFrontend: " unable to obtain stable firefox connection in 60 seconds" blocking merges on MobileFrontend - https://phabricator.wikimedia.org/T132379#2196252 (10Jdlrobson) [18:38:02] 07Browser-Tests, 06Release-Engineering-Team, 10MobileFrontend: " unable to obtain stable firefox connection in 60 seconds" blocking merges on MobileFrontend - https://phabricator.wikimedia.org/T132379#2196267 (10Jdlrobson) [18:45:23] Project beta-scap-eqiad build #97791: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97791/ [18:49:13] 07Blocked-on-RelEng, 06Operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#2196309 (10Dzahn) p:05Triage>03Normal [18:49:28] 07Blocked-on-RelEng, 06Operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#1936415 (10Dzahn) 05Open>03stalled [18:50:01] 07Blocked-on-RelEng, 06Operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#1936415 (10Dzahn) This is stalled as long as we use gitblit. (Or we just upgrade it in place anyways, so we don't wait for that) [18:50:52] 10Beta-Cluster-Infrastructure: beta-scap-eqiad scap fails with CentralIdLookup::resetCache may only be called during bootstrapping unit tests - https://phabricator.wikimedia.org/T132370#2196328 (10thcipriani) This may be related to: https://gerrit.wikimedia.org/r/#/c/270020/ [18:50:55] 07Blocked-on-RelEng, 05Gitblit-Deprecate, 06Operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#2196329 (10Dzahn) [18:54:09] 07Blocked-on-RelEng, 05Gitblit-Deprecate, 06Operations: Phase out antimony.wikimedia.org - https://phabricator.wikimedia.org/T123718#1936415 (10Dzahn) ... or how about just putting gitblit on a jessie VM and shutting this server down? shouldn't have a hard blocker.hm? [18:54:32] 07Blocked-on-RelEng, 05Gitblit-Deprecate, 06Operations: Phase out antimony.wikimedia.org (git.wikimedia.org / gitblit) - https://phabricator.wikimedia.org/T123718#2196342 (10Dzahn) [18:55:20] Project beta-scap-eqiad build #97792: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97792/ [19:05:21] Project beta-scap-eqiad build #97793: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97793/ [19:06:33] 07Blocked-on-RelEng, 05Gitblit-Deprecate, 06Operations: Phase out antimony.wikimedia.org (git.wikimedia.org / gitblit) - https://phabricator.wikimedia.org/T123718#2196410 (10demon) >>! In T123718#2196331, @Dzahn wrote: > ... or how about just putting gitblit on a jessie VM and shutting this server down? shou... [19:11:39] 10Beta-Cluster-Infrastructure: Scap mwscript rebuildLocalisationCache.php fails - https://phabricator.wikimedia.org/T132370#2196425 (10thcipriani) [19:15:22] Project beta-scap-eqiad build #97794: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97794/ [19:20:01] Project beta-update-databases-eqiad build #7783: 04STILL FAILING in 0.62 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7783/ [19:25:23] Project beta-scap-eqiad build #97795: 04STILL FAILING in 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97795/ [19:32:50] RECOVERY - Puppet run on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:33:22] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#2196514 (10Krenair) @yuvipanda: So we need to make a puppet change of some sort? Or we should do something like `salt 'hostname.deployment-prep.eqiad.wmflabs' grains.... [19:35:21] Project beta-scap-eqiad build #97796: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97796/ [19:45:21] Project beta-scap-eqiad build #97797: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97797/ [19:49:27] Project beta-scap-eqiad build #97798: 04STILL FAILING in 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97798/ [19:49:27] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#2196546 (10Krenair) [19:49:47] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#985792 (10Krenair) [19:55:08] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#2196558 (10Krenair) ``` so I could make a hiera page on wikitech for cache-text04 set cluster: cache_text something like https:... [19:55:28] Project beta-scap-eqiad build #97799: 04STILL FAILING in 28 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97799/ [19:55:29] 10Beta-Cluster-Infrastructure: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#2196560 (10Krenair) [20:01:41] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10RESTBase, 06Services: iptables rules on deployment-restbase02 are wrong - https://phabricator.wikimedia.org/T125038#1972853 (10Krenair) @mobrovac: I just went to look into this but found all of the rules are identical on both instances: ```Chain... [20:01:57] 07Blocked-on-RelEng, 05Gitblit-Deprecate, 06Operations: Phase out antimony.wikimedia.org (git.wikimedia.org / gitblit) - https://phabricator.wikimedia.org/T123718#2196572 (10Dzahn) We removed Apache and config and the SSL key from antimony... Gotta fix the Icinga monitoring check next, since it checked Apa... [20:03:41] a couple jobs look stuck: https://integration.wikimedia.org/zuul/ [20:05:31] Project beta-scap-eqiad build #97800: 04STILL FAILING in 27 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97800/ [20:09:27] looks like integration-slave-trusty-1025 is having some issue. [20:13:36] !log killing stuck jobs, marking integration-slave-trusty-1025 as offline temporarily [20:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:15:20] Project beta-scap-eqiad build #97801: 04STILL FAILING in 24 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97801/ [20:16:45] thcipriani: ty [20:16:53] yarp. [20:18:59] !log integration-slave-trusty-1025 horizon console filled with INFO: task jbd2/vda1-8:170 blocked for more than 120 seconds. rebooting [20:19:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:20:01] Project beta-update-databases-eqiad build #7784: 04STILL FAILING in 0.58 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7784/ [20:22:55] 10Beta-Cluster-Infrastructure, 10MediaWiki-Recent-changes: Special:RecentChanges shows still creation and edits of already deleted pages at dewiki beta - https://phabricator.wikimedia.org/T119559#1829667 (10Krenair) Is this still reproducible? I created a page, edited it, and deleted it, and only the deletion... [20:28:35] RECOVERY - SSH on integration-slave-trusty-1025 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 (protocol 2.0) [20:30:14] !log relaunched slave-agent on integration-slave-trusty-1025, back online [20:30:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [20:33:59] Yippee, build fixed! [20:33:59] Project beta-scap-eqiad build #97802: 09FIXED in 9 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/97802/ [20:34:51] yippee! [20:34:55] thanks bd808 [20:35:18] ^ +1 ty! [20:35:20] greg-g: If that's for the rebuildLocalisationCache patch... [20:35:23] partial revert FTW [20:35:25] They're all being reverted now [20:35:33] Roan is just doing it [20:35:47] going to wait until after the branch cut? [20:35:56] Login is completely broken [20:36:03] o_O [20:36:13] Yeah, one of the DI patches broke login [20:36:18] Roan and I could replicate with CA, addshore could without it [20:36:27] I have to wait for the first revert to make it through Jenkins before I can do the second one [20:36:32] Because of conflict weirdness [20:36:41] AdamRevert is a good name [20:36:46] I might use that for kid#2 [20:36:49] RoanKattouw: or cherry pick it onto it! [20:37:13] it just merged [20:37:14] gogoo [20:37:16] Yay [20:37:47] Thanks Gerrit :/ [20:37:56] It let me generate 282788 but that will fail to merge, I need to rebase it [20:38:15] I guess that's still better than forcing me to build that commit by hand though, they actually did this one right [20:46:42] 10Beta-Cluster-Infrastructure, 13Patch-For-Review, 15User-bd808: Scap mwscript rebuildLocalisationCache.php fails - https://phabricator.wikimedia.org/T132370#2196782 (10thcipriani) 05Open>03Resolved a:03bd808 [20:47:34] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #982: 04FAILURE in 21 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/982/ [20:49:37] 10Beta-Cluster-Infrastructure, 10Staging, 10DBA, 03Collab-Archive-2015-2016, and 2 others: Use External Store on Beta Cluster - https://phabricator.wikimedia.org/T95871#2196797 (10Mattflaschen) [20:49:49] 10Beta-Cluster-Infrastructure: Document how and when the configuration is propagated to the beta cluster - https://phabricator.wikimedia.org/T124198#2196800 (10Krenair) 05Open>03Resolved a:03Krenair I tidied and updated that page a little, it sounds like we already have this documentation and the issue wit... [21:00:26] 10Beta-Cluster-Infrastructure: Diamond logstash monitor fills /var/log/apache2 access log - https://phabricator.wikimedia.org/T74175#749228 (10Krenair) ```root@deployment-logstash2:/# grep "server-status" etc/* -r etc/apache2/conf-available/50-server-status.conf:# Only serve /server-status on loopback interface... [21:03:39] 10Beta-Cluster-Infrastructure: Diamond logstash monitor fills /var/log/apache2 access log - https://phabricator.wikimedia.org/T74175#2196854 (10Krenair) (and then ran puppet to put it back how it was) But what do we want to do - disable the ganglia monitor, get rid of access logs more often, or...? [21:04:59] 10Beta-Cluster-Infrastructure: Convert puppetmaster sync cronjob to Jenkins job - https://phabricator.wikimedia.org/T73305#2196856 (10Krenair) [21:13:27] 07Blocked-on-RelEng, 05Gitblit-Deprecate, 06Operations, 13Patch-For-Review: Phase out antimony.wikimedia.org (git.wikimedia.org / gitblit) - https://phabricator.wikimedia.org/T123718#2196931 (10Dzahn) >>! In T123718#2196572, @Dzahn wrote: > We removed Apache and config and the SSL key from antimony... >... [21:14:00] 10Beta-Cluster-Infrastructure, 07Jenkins: Convert puppetmaster sync cronjob to Jenkins job - https://phabricator.wikimedia.org/T73305#2196935 (10Krenair) [21:16:25] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config: Send beta cluster Jenkins alerts to betacluster-alert list - https://phabricator.wikimedia.org/T1125#2196949 (10Krenair) a:03Krenair [21:16:48] (03PS1) 10Alex Monk: Add betacluster-alert list to relevant jobs [integration/config] - 10https://gerrit.wikimedia.org/r/282811 (https://phabricator.wikimedia.org/T1125) [21:17:41] 06Release-Engineering-Team, 06Team-Practices, 10Developer-Relations (Jul-Sep-2016): Developer Summit 2017: Work with TPG and RelEng on solution to event documenting - https://phabricator.wikimedia.org/T132400#2196957 (10Rfarrand) [21:18:04] 10Beta-Cluster-Infrastructure, 07JavaScript, 07Jenkins: Sync pages (gadgets, in CSS and/or JS) from production wikis - https://phabricator.wikimedia.org/T51779#2196970 (10Krenair) [21:20:43] Yippee, build fixed! [21:20:44] Project beta-update-databases-eqiad build #7785: 09FIXED in 42 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/7785/ [21:23:50] !log deployed and reverted oath [21:23:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:28:19] 06Release-Engineering-Team, 15User-greg: Create 1516Q3 quarterly review slide deck - https://phabricator.wikimedia.org/T131954#2196993 (10greg) 05Open>03Resolved [21:36:59] 06Release-Engineering-Team, 15User-greg: schedule mid-Q4 check-ins - https://phabricator.wikimedia.org/T131891#2197016 (10greg) May 2nd and June 6th, at our normal team meeting time @demon will be doing the June 6th one :) [21:37:04] 06Release-Engineering-Team, 15User-greg: schedule mid-Q4 check-ins - https://phabricator.wikimedia.org/T131891#2197018 (10greg) 05Open>03Resolved [21:45:17] 10Beta-Cluster-Infrastructure, 10Monitoring, 07Shinken: Monitor keyholder on deployment-bastion - https://phabricator.wikimedia.org/T111064#2197030 (10Krenair) So we could put some logic in `modules/keyholder/files/check_keyholder` to report success (0) or failure (1) to graphite (like what `modules/puppetma... [21:57:18] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [22:18:27] 10Continuous-Integration-Infrastructure, 10Fundraising Tech Backlog, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: civicrm-buildkit doesn't composer update our wikimedia projects correctly - https://phabricator.wikimedia.org/T104031#2197196 (10DStrine) [22:19:06] 10Continuous-Integration-Config, 10Fundraising Tech Backlog, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Optimize CiviCRM CI job - https://phabricator.wikimedia.org/T91903#2197219 (10DStrine) [22:20:48] 10Continuous-Integration-Config, 10Wikimedia-IEG-grant-review, 10Wikimedia-Wikimania-Scholarships, 13Patch-For-Review: Phase out job 'phpunit' in favor of jobs running 'composer test' - https://phabricator.wikimedia.org/T131917#2197259 (10bd808) >>! In T131917#2183047, @hashar wrote: > I am not proposing a... [22:29:03] 10Beta-Cluster-Infrastructure, 10VisualEditor: [Regression] Search for pages/templates in VE is not working in Beta cluster - https://phabricator.wikimedia.org/T132408#2197341 (10Ryasmeen) [22:31:25] 10Beta-Cluster-Infrastructure, 10VisualEditor: [Regression pre-wmf.21] Search for pages/templates in VE is not working in Beta cluster - https://phabricator.wikimedia.org/T132408#2197356 (10Ryasmeen) [22:37:10] 05Gerrit-Migration, 10Differential: git commit hash remains the same from submitting to after merge - https://phabricator.wikimedia.org/T91420#2197361 (10JanZerebecki) >>! In T91420#2196096, @valhallasw wrote: >> That in turn moves more trust from the ends to a central location. > The way git is used in wikime... [23:29:38] 10Beta-Cluster-Infrastructure, 10Flow, 03Collab-Team-2016-Q4: Run Flow External Store migration in dry-run mode on Beta - https://phabricator.wikimedia.org/T119567#2197536 (10jmatazzoni) [23:54:28] 10Continuous-Integration-Config, 10Fundraising Tech Backlog, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Optimize CiviCRM CI job - https://phabricator.wikimedia.org/T91903#2197663 (10awight) This still takes 85s when the tests themselves are < 10s of work, but I'm fine clo... [23:54:33] 10Continuous-Integration-Config, 10Fundraising Tech Backlog, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Optimize CiviCRM CI job - https://phabricator.wikimedia.org/T91903#2197664 (10awight) 05Open>03Resolved a:03awight [23:54:36] 10Continuous-Integration-Infrastructure, 10Fundraising Tech Backlog, 06Scrum-of-Scrums, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Continuous integration - CiviCRM - https://phabricator.wikimedia.org/T78100#2197666 (10awight)