[00:23:47] Project beta-code-update-eqiad build #176768: 04FAILURE in 47 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176768/ [00:33:44] Project beta-code-update-eqiad build #176769: 04STILL FAILING in 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176769/ [00:35:55] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Backlog): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#3684821 (10MaxSem) [00:35:57] 10Release-Engineering-Team, 10Operations, 10WorkType-NewFunctionality: [Spike] Try out hack ( 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 2 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3684824 (10Paladox) There’s a 2.13 branch https://gerrit.googlesource.com/plugins/lfs/+/stable-2.13 Docs is at https://ger... [00:43:27] 10Release-Engineering-Team (Backlog), 10Deployments, 10Operations, 10Beta-Cluster-reproducible, and 3 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#3684826 (10Jdforrester-WMF) [00:43:43] Project beta-code-update-eqiad build #176770: 04STILL FAILING in 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176770/ [00:44:43] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [00:47:28] !log Trying PHP7 mode on depoyment-prep with https://wikitech.wikimedia.org/w/index.php?diff=1772791 (ping T173786) [00:47:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:47:37] T173786: Convert Wikimedia production HHVM instances to have hhvm.php7.all set true - https://phabricator.wikimedia.org/T173786 [00:48:28] !log reverting [00:48:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:49:29] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1342 bytes in 0.002 second response time [00:49:33] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1975 bytes in 0.039 second response time [00:50:12] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1993 bytes in 0.065 second response time [00:51:52] PROBLEM - App Server Main HTTP Response on deployment-mediawiki05 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1986 bytes in 0.134 second response time [00:53:44] Project beta-code-update-eqiad build #176771: 04STILL FAILING in 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176771/ [00:54:32] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 46780 bytes in 4.104 second response time [00:54:36] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 35274 bytes in 3.282 second response time [00:55:10] RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 47258 bytes in 1.292 second response time [00:56:57] RECOVERY - App Server Main HTTP Response on deployment-mediawiki05 is OK: HTTP OK: HTTP/1.1 200 OK - 46780 bytes in 3.653 second response time [00:59:52] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 2 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3684852 (10demon) We could possibly do FS-based for now, pending work on supporting Swift. [01:01:58] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Scap (Tech Debt Sprint FY201718-Q2), 10ORES, and 2 others: Support git-lfs files in gerrit - https://phabricator.wikimedia.org/T171758#3684856 (10demon) The `/srv/` partition on Gerrit is only 1% full, we've got plenty of space. [01:02:15] PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [43200.0] [01:03:40] Project beta-code-update-eqiad build #176772: 04STILL FAILING in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176772/ [01:13:44] Project beta-code-update-eqiad build #176773: 04STILL FAILING in 44 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176773/ [01:23:48] Project beta-code-update-eqiad build #176774: 04STILL FAILING in 48 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176774/ [01:24:42] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:33:40] Project beta-code-update-eqiad build #176775: 04STILL FAILING in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176775/ [01:43:40] Project beta-code-update-eqiad build #176776: 04STILL FAILING in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176776/ [01:53:41] Project beta-code-update-eqiad build #176777: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176777/ [02:03:41] Project beta-code-update-eqiad build #176778: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176778/ [02:13:42] Project beta-code-update-eqiad build #176779: 04STILL FAILING in 42 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176779/ [02:23:43] Project beta-code-update-eqiad build #176780: 04STILL FAILING in 43 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176780/ [02:33:41] Project beta-code-update-eqiad build #176781: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176781/ [02:40:00] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684901 (10bmansurov) >>! In T178189#3684715, @Legoktm wrote: > Just to clarify, are you trying to commit a binary of chromium into git? I think... [02:41:31] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684902 (10bmansurov) @daemon, ok, let's try git-fat first. [02:43:40] Project beta-code-update-eqiad build #176782: 04STILL FAILING in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176782/ [02:53:41] Project beta-code-update-eqiad build #176783: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176783/ [03:03:41] Project beta-code-update-eqiad build #176784: 04STILL FAILING in 41 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176784/ [03:11:35] fixed now? probably fixed now. [03:13:41] Yippee, build fixed! [03:13:41] Project beta-code-update-eqiad build #176785: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/176785/ [03:13:48] yippee [03:13:56] afk 4 realz [03:37:00] Project mediawiki-core-code-coverage build #3066: 04STILL FAILING in 37 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3066/ [03:45:41] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:15:50] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [04:25:43] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:18:36] 10Gerrit, 10Readers-Web-Backlog, 10Services, 10Patch-For-Review, and 2 others: Temporarily allow pushing large objects - https://phabricator.wikimedia.org/T178189#3684991 (10phuedx) [06:41:49] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [07:41:50] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [08:51:51] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [09:47:42] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [10:22:40] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:49:27] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685123 (10Dvorapa) [11:57:55] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3679445 (10Paladox) ah, this is the same problem we had before. with submodules being updated by gerrit. the date corresponds with the gerri... [11:59:02] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685127 (10Paladox) i think i know how to fix this. [12:07:15] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net, 10Patch-For-Review: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685131 (10Paladox) That ^^ will fix it. It's due to a new security feature in gerrit 2.13. [12:09:38] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net, 10Patch-For-Review: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685132 (10Paladox) to test it, merge the change, then create a change in https://gerrit.wikimedia.org/r/#/admin/projec... [12:18:14] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net, 10Patch-For-Review: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685135 (10Paladox) >>! In T178067#3685109, @valhallasw wrote: > I believe Gerrit automatically updates the references... [12:23:08] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:23:29] (03Abandoned) 10Zfilipin: Popups Selenium tests should run daily targeting beta cluster [integration/config] - 10https://gerrit.wikimedia.org/r/384030 (https://phabricator.wikimedia.org/T177924) (owner: 10Zfilipin) [12:25:07] (03Abandoned) 10Zfilipin: Run mwext-ruby-jessie job for Popups [integration/config] - 10https://gerrit.wikimedia.org/r/384014 (https://phabricator.wikimedia.org/T164479) (owner: 10Zfilipin) [13:04:25] Yippee, build fixed! [13:04:25] Project selenium-Math » chrome,beta,Linux,BrowserTests build #544: 09FIXED in 25 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/544/ [13:04:29] Yippee, build fixed! [13:04:29] Project selenium-Math » firefox,beta,Linux,BrowserTests build #544: 09FIXED in 29 sec: https://integration.wikimedia.org/ci/job/selenium-Math/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/544/ [13:57:52] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [14:02:49] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [14:29:09] 10Browser-Tests-Infrastructure, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Wikidata, and 3 others: Run Wikibase daily browser tests on Jenkins - https://phabricator.wikimedia.org/T167432#3685206 (10zeljkofilipin) I have switched to job back to sauce labs, and the failed... [15:06:42] 10Continuous-Integration-Config, 10Wikidata, 10User-Addshore: Only run npm job on Jenkins for builds of data-values/value-view - https://phabricator.wikimedia.org/T178083#3685248 (10WMDE-leszek) Patch changing the config: https://gerrit.wikimedia.org/r/#/c/383872/ (gerrit-bot didn't pick it up). [15:09:40] (03PS1) 10WMDE-leszek: Only run npm job for changes in mediawiki/extensions/WikibaseJavaScriptApi [integration/config] - 10https://gerrit.wikimedia.org/r/384236 (https://phabricator.wikimedia.org/T178225) [15:10:51] (03CR) 10jerkins-bot: [V: 04-1] Only run npm job for changes in mediawiki/extensions/WikibaseJavaScriptApi [integration/config] - 10https://gerrit.wikimedia.org/r/384236 (https://phabricator.wikimedia.org/T178225) (owner: 10WMDE-leszek) [15:14:10] (03CR) 10WMDE-leszek: "Looks like this should be made not an extension first. Will file a ticket." [integration/config] - 10https://gerrit.wikimedia.org/r/384236 (https://phabricator.wikimedia.org/T178225) (owner: 10WMDE-leszek) [15:36:08] Project mediawiki-core-code-coverage build #3067: 04STILL FAILING in 36 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3067/ [15:40:07] 10Continuous-Integration-Config, 10Wikidata: Only run npm job on Jenkins for builds of mediawiki/extensions/WikibaseJavaScriptApi - https://phabricator.wikimedia.org/T178225#3685258 (10WMDE-leszek) [15:48:12] 10Continuous-Integration-Config, 10Wikidata: Rename mediawiki/extensions/WikibaseJavaScriptApi repo - https://phabricator.wikimedia.org/T178226#3685259 (10WMDE-leszek) [16:13:07] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net, 10Patch-For-Review: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685277 (10valhallasw) 05Open>03Resolved a:03valhallasw Thank you for your help, @Paladox! https://github.com/wi... [16:13:28] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net, 10Patch-For-Review: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685280 (10Paladox) Your welcome :). [16:17:48] 10Gerrit, 10Pywikibot-core, 10Pywikibot-i18n, 10translatewiki.net: Make L10n-bot update commit hash in scripts/i18n again - https://phabricator.wikimedia.org/T178067#3685281 (10Paladox) [17:01:33] 10Continuous-Integration-Config, 10Gerrit, 10Wikidata: Rename mediawiki/extensions/WikibaseJavaScriptApi repo - https://phabricator.wikimedia.org/T178226#3685336 (10Paladox) [18:26:04] PROBLEM - Free space - all mounts on integration-slave-jessie-1002 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1002.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1002.diskspace._srv.byte_percentfree (<22.22%) [19:30:04] 10Release-Engineering-Team (Kanban): Identify Orphaned components/code - https://phabricator.wikimedia.org/T173349#3685466 (10Aklapper) @Jrbranaa: I am curious and I'd like to have more context. Also because we have some stats on https://wikimedia.biterg.io/ (which might not help with this at all, but to judge... [20:13:21] 10Gerrit, 10Developer-Wishlist (2017), 10Upstream: Free-form tagging in gerrit - https://phabricator.wikimedia.org/T37534#3685498 (10Tgr) [20:13:42] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:53:43] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:25:51] (03PS12) 10Umherirrender: Update test config [integration/config] - 10https://gerrit.wikimedia.org/r/380790 [21:31:58] 10Release-Engineering-Team (Kanban), 10Phabricator: Comments section shown on Create Task form - https://phabricator.wikimedia.org/T178235#3685557 (10Luke081515) p:05Triage>03High a:03mmodell Assinging to @mmodell since he normally does that, and was the last one editing this form. Feel free to change if... [21:32:12] huh [21:32:18] twentyafterfour: can you take a look at that? ^ [22:18:30] 10Release-Engineering-Team (Kanban), 10Phabricator: Comments section shown on Create Task form - https://phabricator.wikimedia.org/T178235#3685593 (10SamanthaNguyen)