[00:27:36] 10Gerrit, 06Release-Engineering-Team, 10DBA, 06Operations, 13Patch-For-Review: Gerrit shows HTTP 500 error when pasting extended unicode characters - https://phabricator.wikimedia.org/T145885#2998926 (10Paladox) [01:09:32] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit went down for 1minute on 05/01/17 - https://phabricator.wikimedia.org/T157203#2998958 (10Paladox) [01:11:10] paladox: 05/01/17? [01:11:27] JustBerry see -operations [01:11:40] there are warnnings about cobalt (gerrit) server [01:12:19] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit went down for 1minute on 05/02/17 - https://phabricator.wikimedia.org/T157203#2998985 (10Paladox) [01:12:25] Yes, but I think you meant 05/02/17 or 02/05/17 ;p [01:12:39] paladox: ^^ [01:14:15] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit went down for 1minute on 05/02/17 - https://phabricator.wikimedia.org/T157203#2998988 (10JustBerry) p:05Triage>03High Very important (if not unbreak now). [01:15:46] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit went down for 1minute on 05/02/17 - https://phabricator.wikimedia.org/T157203#2998990 (10Paladox) p:05High>03Triage [01:16:00] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit went down for 1minute on 05/02/17 - https://phabricator.wikimedia.org/T157203#2998958 (10Paladox) p:05Triage>03High [01:17:35] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit went down for 1minute on 05/02/17 - https://phabricator.wikimedia.org/T157203#2998958 (10Paladox) [01:19:15] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit went down for 1minute on 05/02/17 - https://phabricator.wikimedia.org/T157203#2998994 (10JustBerry) Investigate channel logs in #wikimedia-operations as necessary: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/20170205.txt [01:22:31] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Investigate why gerrit went down for 1minute on 05/02/17 and then again 4 minute later - https://phabricator.wikimedia.org/T157203#2998996 (10Paladox) [01:37:51] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Investigate why gerrit went down for 1minute on 05/02/17 and then again 4 minute later - https://phabricator.wikimedia.org/T157203#2999000 (10Paladox) p:05High>03Unbreak! Setting unbreak as it appears loading gerrit changes is taking longer then... [02:09:56] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Investigate why gerrit went down for 1minute on 05/02/17 and then again 4 minute later - https://phabricator.wikimedia.org/T157203#2999010 (10Paladox) [02:10:47] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Investigate why gerrit went down for 1minute on 05/02/17 and then again 4 minute later - https://phabricator.wikimedia.org/T157203#2998958 (10Paladox) [02:48:37] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Investigate why gerrit went down for 1minute on 05/02/17 (dd/mm/yy) and then again 4 minute later - https://phabricator.wikimedia.org/T157203#2999067 (10JustBerry) [04:05:15] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #287: 04FAILURE in 9 min 15 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/287/ [04:19:26] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #287: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/287/ [05:25:17] 10Gerrit, 06Release-Engineering-Team, 06Operations: Gerrit: Investigate why gerrit went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999157 (10Jay8g) [07:43:32] 10Gerrit, 10Developer-Wishlist (2017), 07Documentation: Enable and document "WIP" workflow status in Gerrit - https://phabricator.wikimedia.org/T135245#2999176 (10Tgr) [07:45:01] 10Gerrit, 06Developer-Relations, 10Developer-Wishlist (2017): Implement a way to bring GitHub pull requests into gerrit - https://phabricator.wikimedia.org/T37497#2999177 (10Tgr) [08:58:26] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why colbolt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999197 (10Peachey88) p:05Unbreak!>03Triage [08:58:42] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2998958 (10Peachey88) [09:00:43] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2998958 (10Peachey88) > Setting unbreak as it appears loading gerrit changes is taking longer then usual to load. Seems f... [09:17:34] 10Deployment-Systems, 10Developer-Wishlist (2017): Expose php warnings in mediawiki-config more visibly - https://phabricator.wikimedia.org/T87447#2999216 (10Tgr) 05Open>03Resolved a:03Tgr This seems to be thoroughly solved by logstash and the test/canary hosts. > Most links in https://wikitech.wikimedi... [09:54:57] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: mediawiki-extensions-qunit-jessie tests are failing - https://phabricator.wikimedia.org/T153597#2999242 (10Tgr) This task is about a specific issue that has been fixed. I don't see much point in adding it to the dev... [11:44:34] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999349 (10Paladox) >>! In T157203#2999206, @Peachey88 wrote: >> Setting unbreak as it appears loading gerrit changes is t... [11:57:54] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999354 (10Paladox) Cobalt semed to be having higher then normal cpu levels, it's showing the system using a lot of cpu. (... [11:59:55] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999355 (10Paladox) p:05Triage>03High Changing to high as needs investigation as soon as possible. @Peachey88 if you c... [12:01:42] Yippee, build fixed! [12:01:43] Project selenium-RelatedArticles » chrome,beta-mobile,Linux,contintLabsSlave && UbuntuTrusty build #300: 09FIXED in 42 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/300/ [12:02:14] Project selenium-RelatedArticles » chrome,beta-desktop,Linux,contintLabsSlave && UbuntuTrusty build #300: 04FAILURE in 1 min 14 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/300/ [15:07:53] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999445 (10Dzahn) > [01:06:17] PROBLEM - configured eth on cobalt is CRITICAL: CHECK_NRPE: Socket timeout aft... [15:11:44] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999446 (10Paladox) >>! In T157203#2999445, @Dzahn wrote: >> [01:06:17] PROBLEM - configured eth on cobalt is... [18:14:56] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [19:38:07] 10Gerrit, 10MediaWiki-extensions-General-or-Unknown, 06Repository-Admins: Mark Vine extension as a read-only repository on Gerrit - https://phabricator.wikimedia.org/T157224#2999638 (10Paladox) [20:50:07] 10Gerrit, 10MediaWiki-extensions-General-or-Unknown, 06Repository-Admins: Mark Vine extension as a read-only repository on Gerrit - https://phabricator.wikimedia.org/T157224#2999707 (10Florian) a:03Florian [20:51:57] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [20:53:19] (03PS1) 10Florianschmidtwelzow: Remove tests for Vine extension [integration/config] - 10https://gerrit.wikimedia.org/r/336127 (https://phabricator.wikimedia.org/T157224) [20:53:36] 10Gerrit, 10MediaWiki-extensions-General-or-Unknown, 06Repository-Admins, 13Patch-For-Review: Mark Vine extension as a read-only repository on Gerrit - https://phabricator.wikimedia.org/T157224#2999711 (10Florian) Marked as read-only in gerrit. Waiting for CI change to be merged. [20:56:54] (03CR) 10Paladox: [C: 031] Remove tests for Vine extension [integration/config] - 10https://gerrit.wikimedia.org/r/336127 (https://phabricator.wikimedia.org/T157224) (owner: 10Florianschmidtwelzow) [21:27:41] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999739 (10hashar) The huge User CPU spike around 21:00UTC is me doing maintenance on Zuul git repositories. Went on scand... [22:47:35] 10Gerrit, 06Release-Engineering-Team, 06Operations: Investigate why cobalt went down for 1 minute on Feburary 5 and then again 4 minutes later - https://phabricator.wikimedia.org/T157203#2999835 (10Paladox) >>! In T157203#2999739, @hashar wrote: > The huge User CPU spike around 21:00UTC is me doing maintenan... [23:17:44] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: phpunit 4.8 no longer supported - https://phabricator.wikimedia.org/T157239#2999864 (10Reedy) [23:18:00] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: phpunit 4.8 no longer supported - https://phabricator.wikimedia.org/T157239#2999878 (10Reedy) p:05Triage>03Low [23:35:48] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: phpunit 4.8 no longer supported - https://phabricator.wikimedia.org/T157239#2999864 (10Paladox) @Reedy oh-unit 5.7 requires php 5.6, but we will need to support php 5.5 as 1.27 supports php 5.5. [23:36:50] 10Continuous-Integration-Config, 10MediaWiki-Unit-tests: phpunit 4.8 no longer supported - https://phabricator.wikimedia.org/T157239#2999884 (10Paladox) Also we need 4.8 to support php 5.5. We could have migrated to 5.7 but they support php 5.6+ not php 5.5+