[00:07:25] PROBLEM - Host deployment-cumin is DOWN: CRITICAL - Host Unreachable (172.16.5.1) [06:39:44] 10Phabricator, 10User-DannyS712: Herald: Rules request for DannyS712 - https://phabricator.wikimedia.org/T221574 (10DannyS712) [07:49:47] 10Release-Engineering-Team (Backlog), 10Readers-Web-Backlog, 10Browser-Tests, 10User-zeljkofilipin: Have a discussion around Minerva selenium browser test architecture - https://phabricator.wikimedia.org/T220755 (10Jdlrobson) p:05Triage→03High [08:29:41] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10Krenair) [08:29:47] 10Beta-Cluster-Infrastructure, 10Acme-chief, 10Patch-For-Review: Write designate integration script for certcentral DNS challenges - https://phabricator.wikimedia.org/T206922 (10Krenair) 05Open→03Resolved [08:36:22] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10Krenair) 05Open→03Resolved [08:36:34] 10Beta-Cluster-Infrastructure: Beta eswikibooks certificate issues - https://phabricator.wikimedia.org/T199387 (10Krenair) 05Stalled→03Resolved a:03Krenair [08:36:38] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Patch-For-Review: Get letsencrypt wildcard cert for *.beta.wmflabs.org domains - https://phabricator.wikimedia.org/T182927 (10Krenair) [09:31:18] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [10:22:04] !log ores:060fc37 going beta [10:22:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:42:46] 10Beta-Cluster-Infrastructure, 10DNS, 10Operations, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10MoritzMuehlenhoff) All bust... [11:52:40] (03PS1) 10Alexandros Kosiaris: Create/Publish OCI images for termbox [integration/config] - 10https://gerrit.wikimedia.org/r/505755 (https://phabricator.wikimedia.org/T220402) [11:57:57] (03CR) 10Hashar: [C: 03+2] [CrawlableAllPages] Add quibble [integration/config] - 10https://gerrit.wikimedia.org/r/505270 (owner: 10Umherirrender) [11:59:02] (03CR) 10Hashar: [C: 03+2] Create/Publish OCI images for termbox [integration/config] - 10https://gerrit.wikimedia.org/r/505755 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris) [11:59:24] (03Merged) 10jenkins-bot: [CrawlableAllPages] Add quibble [integration/config] - 10https://gerrit.wikimedia.org/r/505270 (owner: 10Umherirrender) [12:00:35] (03CR) 10Hashar: [C: 03+2] [ApiFeatureUsage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505452 (owner: 10Umherirrender) [12:00:47] (03Merged) 10jenkins-bot: Create/Publish OCI images for termbox [integration/config] - 10https://gerrit.wikimedia.org/r/505755 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris) [12:01:35] (03CR) 10Hashar: [C: 03+2] [CreditsSource] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505450 (owner: 10Umherirrender) [12:02:00] (03Merged) 10jenkins-bot: [ApiFeatureUsage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505452 (owner: 10Umherirrender) [12:03:30] (03Merged) 10jenkins-bot: [CreditsSource] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505450 (owner: 10Umherirrender) [12:03:33] (03CR) 10Hashar: [C: 03+2] [ContactPage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505445 (owner: 10Umherirrender) [12:04:56] (03Merged) 10jenkins-bot: [ContactPage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505445 (owner: 10Umherirrender) [12:30:57] (03CR) 10Hashar: "The extension now uses the Zuul template extension-quibble which indeed relies on vendor.git" [integration/config] - 10https://gerrit.wikimedia.org/r/434011 (https://phabricator.wikimedia.org/T191537) (owner: 10WMDE-leszek) [12:53:32] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [14:31:13] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 10 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10Jdlrobson) In https://gerrit.wikimedia.o... [14:41:29] !log Shut down deployment-ms-be03 and deployment-ms-be04 T218729 [14:41:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:41:38] T218729: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 [14:42:56] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [14:44:28] PROBLEM - Host deployment-ms-be03 is DOWN: CRITICAL - Host Unreachable (172.16.5.51) [14:45:19] PROBLEM - Host deployment-ms-be04 is DOWN: CRITICAL - Host Unreachable (172.16.4.129) [14:45:22] 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair) [15:33:58] !log merging the 2.15.13 release into stable-2.15 following https://wikitech.wikimedia.org/wiki/Gerrit#Update_our_repository [15:33:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:42:26] 10Beta-Cluster-Infrastructure, 10serviceops: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Andrew) [15:47:00] 10Beta-Cluster-Infrastructure, 10serviceops: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Andrew) [16:07:42] (03PS1) 10Lucas Werkmeister (WMDE): Add WikibaseSchema extension to make-wmf-branch [tools/release] - 10https://gerrit.wikimedia.org/r/505808 (https://phabricator.wikimedia.org/T221648) [16:10:11] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 10 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10hashar) >>! In T199939#5130664, @Jdlrobs... [16:11:43] 10Continuous-Integration-Config, 10MediaWiki-Core-Testing, 10MediaWiki-extensions-MultimediaViewer, 10MobileFrontend, and 10 others: Audit tests/selenium/LocalSettings.php file aiming at possibly deprecating the feature - https://phabricator.wikimedia.org/T199939 (10hashar) Adding Minerva is T202030 [16:18:34] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 35.71% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:26:51] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [17:01:51] 10Continuous-Integration-Config, 10translatewiki.net, 10I18n: Fix mediawiki-i18n-check-docker on non-mw repos - https://phabricator.wikimedia.org/T221672 (10Umherirrender) [17:31:44] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Gerrit thread use GC thrashing - https://phabricator.wikimedia.org/T221026 (10thcipriani) `changeid_projects` cache is, today, looking like it's in poor shape: ` Name |Entries | AvgGet |Hit Ratio|... [17:32:35] (03PS1) 10Umherirrender: [ExternalGuidance] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505834 [17:35:57] (03PS1) 10Umherirrender: [FeaturedFeeds] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505838 [17:41:24] (03PS1) 10Umherirrender: [FundraisingTranslateWorkflow] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505841 [17:43:33] almost every gerrit interaction takes 3-10 seconds to complete. Might have something to do with the recent upgrade? [17:43:48] opening links, saving comments etc. [17:44:49] (03PS1) 10Umherirrender: [GlobalBlocking] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505843 [17:48:19] (03PS1) 10Umherirrender: [GlobalCssJs] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/505845 [17:49:05] Krinkle it appears threads have sky rocketed [17:49:15] meaning we are experincing partial outages. [18:08:25] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10Krinkle) This is still the slowest step of `beta-scap-eqiad`. Taking 9 minutes of the total 10 minutes and 5 se... [18:30:01] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Scap: On deployment-prep scap cache_git_info takes 12 minutes (that is too slow) - https://phabricator.wikimedia.org/T204762 (10Krenair) >>! In T204762#4595548, @thcipriani wrote: > Takes a while on beta because of all the extensions (plus... [18:37:54] PROBLEM - Puppet errors on integration-castor03 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [3.0] [18:38:12] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 3.37% of data above the critical threshold [3.0] [18:40:53] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 4.44% of data above the critical threshold [3.0] [18:52:20] PROBLEM - Puppet errors on webperformance is CRITICAL: CRITICAL: 3.33% of data above the critical threshold [3.0] [18:52:44] PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: CRITICAL: 5.56% of data above the critical threshold [3.0] [18:52:46] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: CRITICAL: 4.44% of data above the critical threshold [3.0] [18:53:44] PROBLEM - Puppet errors on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 5.56% of data above the critical threshold [3.0] [18:54:29] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 2.25% of data above the critical threshold [3.0] [18:57:54] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 2.22% of data above the critical threshold [3.0] [18:58:47] PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 2.22% of data above the critical threshold [3.0] [18:59:04] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 4.49% of data above the critical threshold [3.0] [19:02:47] (03PS3) 10Jforrester: Skip php70 tests for wmf branches; we don't run it in prod [integration/config] - 10https://gerrit.wikimedia.org/r/502343 [19:03:07] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 4.49% of data above the critical threshold [3.0] [19:05:37] (03CR) 10jerkins-bot: [V: 04-1] Skip php70 tests for wmf branches; we don't run it in prod [integration/config] - 10https://gerrit.wikimedia.org/r/502343 (owner: 10Jforrester) [20:51:30] 10Scap, 10Wikidata, 10Wikidata-Query-Service: scap service restarts for WDQS are inconsistent - https://phabricator.wikimedia.org/T221709 (10Smalyshev) [21:24:08] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [2.0] [21:24:29] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [2.0] [21:27:52] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [2.0] [21:28:09] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [2.0] [21:28:47] RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [2.0] [21:32:57] RECOVERY - Puppet errors on integration-castor03 is OK: OK: Less than 1.00% above the threshold [2.0] [21:38:13] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [2.0] [21:40:54] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [2.0] [21:47:42] RECOVERY - Puppet errors on integration-slave-jessie-1004 is OK: OK: Less than 1.00% above the threshold [2.0] [21:48:43] RECOVERY - Puppet errors on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [2.0] [21:52:20] RECOVERY - Puppet errors on webperformance is OK: OK: Less than 1.00% above the threshold [2.0] [21:52:44] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [2.0] [22:53:59] 10Phabricator: Rename account Wiki-1776 to Hispano76 - https://phabricator.wikimedia.org/T221725 (10Wiki-1776) [22:58:42] 10Phabricator: Rename account Wiki-1776 to Hispano76 - https://phabricator.wikimedia.org/T221725 (10Wiki-1776)