[00:17:31] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:28:43] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [00:32:33] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [00:52:42] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:00:45] PROBLEM - Puppet errors on deployment-puppetdb01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:13:25] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:15:46] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [01:22:43] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:28:26] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [01:28:42] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [01:30:44] RECOVERY - Puppet errors on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:51:45] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:23:54] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [02:53:45] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:01:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [03:13:08] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [03:43:06] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [03:58:26] Project selenium-MultimediaViewer » firefox,mediawiki,Linux,BrowserTests build #455: 04FAILURE in 2 min 25 sec: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=mediawiki,PLATFORM=Linux,label=BrowserTests/455/ [03:58:42] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:03:54] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:49:06] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [04:49:44] PROBLEM - Puppet errors on deployment-salt02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:04:44] RECOVERY - Puppet errors on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:39:20] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [05:44:00] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [05:54:36] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:09:02] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [06:14:43] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:24:04] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [06:34:18] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:34:35] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:42:47] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:54:46] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:17:47] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:29:43] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:39:41] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [08:05:41] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:16:27] RECOVERY - Puppet staleness on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [3600.0] [08:39:40] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:47:57] (03PS1) 10Hashar: rm operations/debs/python-prometheus-snmp-exporter [integration/config] - 10https://gerrit.wikimedia.org/r/365561 [08:50:30] (03PS2) 10Hashar: Rename operations/debs/python-prometheus-snmp-exporter [integration/config] - 10https://gerrit.wikimedia.org/r/365561 [08:53:30] (03CR) 10Hashar: [C: 032] Rename operations/debs/python-prometheus-snmp-exporter [integration/config] - 10https://gerrit.wikimedia.org/r/365561 (owner: 10Hashar) [08:54:40] RECOVERY - Puppet errors on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [08:54:41] (03CR) 10Hashar: [C: 032] Extenson:NSFileRepo add composer unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/364697 (owner: 10Robert Vogel) [08:55:20] (03CR) 10Hashar: [C: 032] Archive AjaxLogin extension [integration/config] - 10https://gerrit.wikimedia.org/r/363774 (https://phabricator.wikimedia.org/T169670) (owner: 10Florianschmidtwelzow) [08:55:42] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [08:56:17] (03Merged) 10jenkins-bot: Rename operations/debs/python-prometheus-snmp-exporter [integration/config] - 10https://gerrit.wikimedia.org/r/365561 (owner: 10Hashar) [08:56:19] (03Merged) 10jenkins-bot: Extenson:NSFileRepo add composer unit tests [integration/config] - 10https://gerrit.wikimedia.org/r/364697 (owner: 10Robert Vogel) [08:56:22] (03Merged) 10jenkins-bot: Archive AjaxLogin extension [integration/config] - 10https://gerrit.wikimedia.org/r/363774 (https://phabricator.wikimedia.org/T169670) (owner: 10Florianschmidtwelzow) [08:58:48] (03CR) 10Hashar: [C: 032] Archive the mediawiki/extensions/Pickle [integration/config] - 10https://gerrit.wikimedia.org/r/363614 (https://phabricator.wikimedia.org/T167610) (owner: 10Florianschmidtwelzow) [08:59:03] 10Gerrit, 10MediaWiki-Extensions-Pickle, 10Repository-Admins, 10Patch-For-Review: Archive Pickle repo at Gerrit - https://phabricator.wikimedia.org/T167610#3442861 (10hashar) [08:59:47] (03Merged) 10jenkins-bot: Archive the mediawiki/extensions/Pickle [integration/config] - 10https://gerrit.wikimedia.org/r/363614 (https://phabricator.wikimedia.org/T167610) (owner: 10Florianschmidtwelzow) [09:25:47] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:48:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [09:53:41] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [10:04:35] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Upstream: Upgrade etsy phan 0.7..0.8 (was mediawiki-core-php70-phan-jessie requested PHP extension sqlite3 has the wrong version) - https://phabricator.wikimedia.org/T169904#3443029 (10hashar) [10:04:40] 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review: Wikibase repo/Wikibase.i18n.namespaces.php fails to pass phan 0.8 - https://phabricator.wikimedia.org/T169980#3443026 (10hashar) 05Open>03Resolved a:03hashar https://gerrit.wikimedia.org/r/... [10:04:49] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Upstream: Upgrade etsy phan 0.7..0.8 (was mediawiki-core-php70-phan-jessie requested PHP extension sqlite3 has the wrong version) - https://phabricator.wikimedia.org/T169904#3412134 (10hashar) [10:05:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Upstream: Upgrade etsy phan 0.7..0.8 (was mediawiki-core-php70-phan-jessie requested PHP extension sqlite3 has the wrong version) - https://phabricator.wikimedia.org/T169904#3443037 (10hashar) 05Open>03Re... [10:08:41] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:37] 10Continuous-Integration-Infrastructure, 10Cloud-Services, 10Beta-Cluster-reproducible, 10Puppet: New instances attached to a role::puppetmaster::standalone Puppetmaster need manual changes after switching from the default Puppetmaster - https://phabricator.wikimedia.org/T148929#3443054 (10hashar) Thank yo... [10:13:11] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, and 8 others: Get CRM CI php version the same as live - https://phabricator.wikimedia.org/T169806#3409147 (10hashar) Well done @Eileenmcnaughton and @Ejegg ! [10:30:45] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:32:29] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:35:39] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:45:19] 10Gerrit, 10MediaWiki-Extensions-Pickle, 10Repository-Admins, 10Patch-For-Review: Archive Pickle repo at Gerrit - https://phabricator.wikimedia.org/T167610#3443135 (10Florian) 05Open>03Resolved [10:59:32] PROBLEM - Puppet errors on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [11:00:20] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [11:05:37] RECOVERY - Puppet errors on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:32] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:22:28] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:43:29] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:54:18] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:29:18] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:36:42] twentyafterfour upstream managed to reproduce my bug with php segfaulting (mariadb) [12:36:43] https://jira.mariadb.org/browse/MDEV-13317?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel [12:39:56] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [12:43:28] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:44:43] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:05:42] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:10:45] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:22:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [13:24:39] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:30:42] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [13:30:58] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [13:33:21] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:49:37] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:49:41] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [13:51:00] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [13:52:42] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [13:58:25] RECOVERY - Puppet errors on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:01:40] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [14:17:22] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3443722 (10Osnard) @Paladox I test only with MediaWikiCore@REL1_27. Can you maybe help me with another issue? At [1] jenkins gives "-1" because an image file th... [14:19:53] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [14:20:01] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3443728 (10Paladox) @Osnard hi, did you do any changes to the css file? Or any resource loader changes? [14:23:15] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:24:39] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:30] 10Continuous-Integration-Infrastructure, 10Cloud-VPS, 10Nodepool, 10Patch-For-Review: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3443790 (10Andrew) Since https://gerrit.wikimedia.org/r/364901 there haven't been any obvious timeouts in contintclou... [14:29:29] I'm speeding nodepool back up (not to where it was last week but to where it was before all the timeouts.) Please let me know if you see weird behavior. [14:38:57] 10Scap (Scap3-Adoption-Phase1), 10scap2, 10Cassandra, 10RESTBase-Cassandra: Deploy logstash logback encoder with scap3 - https://phabricator.wikimedia.org/T116340#3443851 (10Eevans) >>! In T116340#3427971, @greg wrote: > It's within the Cassandra dir on tin: > > > ``` > gjg@tin:/srv/deployment/cassandra$... [14:43:18] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [14:48:43] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:51:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, 10Patch-For-Review: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3433128 (10hashar) [14:55:11] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:56:42] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [14:57:48] PROBLEM - Puppet errors on deployment-prometheus01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:00:42] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:00:54] (03PS2) 10Ejegg: Get rid of npm test for CiviCRM [integration/config] - 10https://gerrit.wikimedia.org/r/364804 [15:03:30] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:08:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, 10Patch-For-Review: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3443947 (10hashar) For the record, changes to `nodepool.yaml` are... [15:10:16] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:12:42] PROBLEM - Puppet errors on deployment-mathoid is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:23] PROBLEM - Puppet errors on deployment-mcs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:13:37] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:15:47] PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:27:31] PROBLEM - Puppet errors on deployment-eventlogging03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:30] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:52:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, 10Patch-For-Review: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3444210 (10hashar) There is too many metrics under `servers.labco... [15:54:41] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:56:02] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3444253 (10thcipriani) [15:56:04] 10Scap (Scap3-Adoption-Phase1), 10Cassandra, 10RESTBase-Cassandra, 10Services (next): Deploy cassandra metrics collector via scap3 - https://phabricator.wikimedia.org/T137371#3444252 (10thcipriani) [15:57:38] 10Scap (Scap3-Adoption-Phase1): Migrate dropwizard/metrics to scap3 - https://phabricator.wikimedia.org/T170839#3444267 (10thcipriani) [15:57:58] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#2101252 (10thcipriani) [15:58:00] 10Scap (Scap3-Adoption-Phase1): Migrate dropwizard/metrics to scap3 - https://phabricator.wikimedia.org/T170839#3444279 (10thcipriani) [15:59:13] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): blubber: produces empy ENV statements - https://phabricator.wikimedia.org/T170285#3444284 (10dduvall) 05Open>03Resolved [16:01:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Cloud-VPS, 10Nodepool, 10Patch-For-Review: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3444291 (10hashar) And on the [[ https://grafana-admin.wikimedia.... [16:05:05] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban: deployment-kafka01 out of disk space - https://phabricator.wikimedia.org/T170523#3444318 (10Nuria) [16:06:23] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban: deployment-kafka01 out of disk space - https://phabricator.wikimedia.org/T170523#3434437 (10Nuria) Any of us during ops week should be able to do it, ping @Ottomata , what is the best way to free space [16:07:02] 10Beta-Cluster-Infrastructure: deployment-eventlogging03 out of disk space - https://phabricator.wikimedia.org/T170522#3434423 (10Nuria) I was dropping log database [16:07:18] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban: deployment-eventlogging03 out of disk space - https://phabricator.wikimedia.org/T170522#3444339 (10Nuria) [16:11:35] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban: deployment-kafka01 out of disk space - https://phabricator.wikimedia.org/T170523#3444383 (10Ottomata) Look for what is causing the used space. Looks like /var/log/daemon.log* was really huge. Not really sure why, I see lots of errors about puppet failing, bu... [16:15:20] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [16:18:03] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:37] PROBLEM - Puppet errors on deployment-zookeeper02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:21:23] RECOVERY - Puppet staleness on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [3600.0] [16:21:33] 10Browser-Tests-Infrastructure, 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 6 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3444451 (10greg) [16:21:42] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:25:20] RECOVERY - Free space - all mounts on deployment-kafka01 is OK: OK: All targets OK [16:28:50] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#3444508 (10mmodell) [16:28:59] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10Patch-For-Review, 10WorkType-NewFunctionality: Deploy phabricator to phab2001.codfw.wmnet - https://phabricator.wikimedia.org/T137928#2559060 (10mmodell) [16:31:43] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:32:33] 10Release-Engineering-Team (Kanban), 10Phabricator (Upstream), 10Upstream: Custom fields for task subtypes fatal on pages > 1 with: Query "ManiphestTaskQuery" failed to return a value from getPagingValueMap() - https://phabricator.wikimedia.org/T168556#3444518 (10mmodell) a:05mmodell>03None [16:33:01] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3444523 (10greg) [16:34:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Set up experimental Docker CI slave - https://phabricator.wikimedia.org/T150502#3444530 (10thcipriani) a:05dduvall>03thcipriani [16:38:41] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10WorkType-NewFunctionality: Configure phabricator clustering for daemons and repositories - https://phabricator.wikimedia.org/T143175#3444593 (10greg) [16:40:06] 10Release-Engineering-Team (Kanban), 10Phabricator, 10Availability, 10WorkType-NewFunctionality: Configure phabricator clustering for daemons and repositories - https://phabricator.wikimedia.org/T143175#3444595 (10mmodell) a:05mmodell>03None [16:40:08] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Evaluate deployment-pipeline-jenkins plugin - https://phabricator.wikimedia.org/T159863#3444599 (10dduvall) [16:40:10] 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: Build testing k8s setup in ci-staging - https://phabricator.wikimedia.org/T159864#3444597 (10dduvall) 05Open>03declined A k8s cluster was successfully set up but it was only used for a brief experiment. The k8s puppet modules ha... [16:40:20] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban: deployment-kafka01 out of disk space - https://phabricator.wikimedia.org/T170523#3444604 (10Nuria) The cause is likely logging of mediawiki revision-create events last week [16:40:38] 10Release-Engineering-Team (Backlog), 10Phabricator, 10Availability, 10WorkType-NewFunctionality: Configure phabricator clustering for daemons and repositories - https://phabricator.wikimedia.org/T143175#2559491 (10mmodell) [16:41:58] 10Deployment-Systems, 10Release-Engineering-Team (Kanban): Automate the recurring management of wikitech:Deployments and phab:#train_deployments - https://phabricator.wikimedia.org/T114488#3444613 (10mmodell) This is still progressing: Current status is I got down a rabbit hole learning //lua + scribunto//. [16:42:27] 10Release-Engineering-Team (Watching / External), 10Wikidata, 10Story: [Story] Use composer-merge-plugin to include Wikidata components in mediawiki-vendor - https://phabricator.wikimedia.org/T95663#3444616 (10Legoktm) @addshore and I have been discussing this lately and he uploaded a giant proof of concept... [16:43:13] 10Release-Engineering-Team (Kanban), 10Operations, 10Phabricator, 10Traffic: Verify that the codfw lvs is configured correctly for Phabricator - https://phabricator.wikimedia.org/T168699#3444618 (10mmodell) Still waiting on merge of https://gerrit.wikimedia.org/r/#/c/355869/ [16:44:43] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:46:00] 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, 10RelatedArticles: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3444624 (10Jdlrobson) [16:47:29] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3218909 (10greg) Status? [16:55:38] RECOVERY - Puppet errors on deployment-zookeeper02 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:19] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [17:00:43] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [17:20:50] (03CR) 10Bearloga: "I uploaded a patch to ortiz (https://gerrit.wikimedia.org/r/#/c/364000/). Was the ortiz-test-jessie job supposed to trigger?" [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [17:28:00] PROBLEM - Free space - all mounts on deployment-eventlogging03 is CRITICAL: CRITICAL: deployment-prep.deployment-eventlogging03.diskspace.root.byte_percentfree (<100.00%) [17:35:43] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [18:01:29] hey, can someone take a look at https://gerrit.wikimedia.org/r/#/c/365198/ please? [18:12:17] MaxSem: lgtm, I can deploy if you can check it once I do [18:12:43] thcipriani, thanks - ready when you are [18:12:57] * thcipriani does [18:13:13] (03CR) 10Thcipriani: [C: 032] Restore ArticleCreationWorkflow [integration/config] - 10https://gerrit.wikimedia.org/r/365198 (https://phabricator.wikimedia.org/T170354) (owner: 10MaxSem) [18:15:33] (03Merged) 10jenkins-bot: Restore ArticleCreationWorkflow [integration/config] - 10https://gerrit.wikimedia.org/r/365198 (https://phabricator.wikimedia.org/T170354) (owner: 10MaxSem) [18:20:27] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/365198 [18:20:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:20:50] ^ MaxSem should be live now [18:22:42] hey seems to have triggered some tests :) [18:23:01] yup, waiting for them to finish [18:25:03] seems to work (I need to make it pass though) [18:25:08] thanks, thcipriani ! [18:25:20] nice :) [18:25:22] yw [18:28:04] twentyafterfour i wonder why https://phabricator.wikimedia.org/guides/quickstart/ this takes a long time to load (hits php timeout) [18:28:12] i would have though it's a quick start guide [18:28:16] so would not load alot [18:28:24] T167221 [18:28:24] T167221: Phabricator max exec time exceeded when going to https://phabricator.wikimedia.org/guides/quickstart/ - https://phabricator.wikimedia.org/T167221 [18:29:01] paladox: it runs a bunch of setup checks within the guide [18:29:18] oh i see. (is it meant only for the admin?) [18:51:14] thcipriani, can https://integration.wikimedia.org/ci/job/mwext-testextension-hhvm-jessie/11099/console be caused by `composer install` not being run? [18:51:35] (03PS1) 10Thcipriani: Remove MinervaNeue from the MobileFrontend dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/365678 (https://phabricator.wikimedia.org/T170624) [18:53:21] (03CR) 10Jdlrobson: [C: 04-1] "This is going to break lots of other jobs (including on RelatedArticles)..." [integration/config] - 10https://gerrit.wikimedia.org/r/365678 (https://phabricator.wikimedia.org/T170624) (owner: 10Thcipriani) [18:53:31] MaxSem that change was introduced in one of the codesniffer version [18:53:34] will check which one [18:54:05] (03Abandoned) 10Thcipriani: Remove MinervaNeue from the MobileFrontend dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/365678 (https://phabricator.wikimedia.org/T170624) (owner: 10Thcipriani) [18:54:40] https://phabricator.wikimedia.org/diffusion/MCSN/browse/master/HISTORY.md;6d804d121d137237e6b9936192a51dfae52479fc$5 [18:54:44] https://phabricator.wikimedia.org/rMCSNecaffdffaa49d35bd43001a774d246d6600fb6c5 [18:54:47] MaxSem ^^ [18:55:12] Upgrade codesniffer to 0.8.1 [18:55:16] thanks! [18:56:14] Your welcome :) [19:17:34] thcipriani: hey [19:17:43] jdlrobson: yo [19:17:43] thcipriani: so now i have https://phabricator.wikimedia.org/T170624 [19:17:47] want a quick and dirty fix for wmf9? [19:19:20] https://gerrit.wikimedia.org/r/#/c/365647/ < thcipriani sorry this is becoming so messy [19:19:36] https://gerrit.wikimedia.org/r/#/c/365647/ has 2 dependencies now which should let it pass [19:20:07] 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, 10RelatedArticles, and 2 others: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3445681 (10Jdlrobson) p:05Triage>03High This is causing issues with the train so I'v... [19:20:13] 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, 10RelatedArticles, and 2 others: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3445684 (10Jdlrobson) a:03Jdlrobson [19:20:21] 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, 10RelatedArticles, and 3 others: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3437378 (10Jdlrobson) [19:20:33] jdlrobson: that will work, thanks for working with me, I really didn't want to have to kick zuul to backport stuff [19:20:46] thcipriani: just waiting for jenkins confirmation [19:26:28] thcipriani: got it [19:26:32] https://gerrit.wikimedia.org/r/#/c/365647/ is ready to ship (with deps) [19:26:52] nice :) [19:27:06] * thcipriani goes a'mergin' [19:31:35] 10Release-Engineering-Team, 10MinervaNeue, 10Reading-Web-Backlog, 10RelatedArticles, and 3 others: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3445755 (10Jdlrobson) I've worked out the problematic test: https://gerrit.wikimedia.org... [19:35:22] 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, 10Security-General: setup releases1001.eqiad.wmnet (was: setup mwreleases1001) - https://phabricator.wikimedia.org/T164030#3445766 (10Dzahn) @greg I blocked him by saying we need to test the uploading (lots of history with having to deb... [19:40:55] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:55:21] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban: deployment-eventlogging03 out of disk space - https://phabricator.wikimedia.org/T170522#3434423 (10Nuria) a:03Nuria [19:55:55] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #13: 04FAILURE in 1 hr 6 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/13/ [19:55:55] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:55:57] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban: deployment-eventlogging03 out of disk space - https://phabricator.wikimedia.org/T170522#3434423 (10Nuria) I have dropped logs in var/log and also srv/eventlogging but still there is a lot of space consumed by db that i have not been able to drop [19:58:42] (03CR) 10Hashar: [C: 031] "> I uploaded a patch to ortiz (https://gerrit.wikimedia.org/r/#/c/364000/). Was the ortiz-test-jessie job supposed to trigger?" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/362309 (https://phabricator.wikimedia.org/T153856) (owner: 10Hashar) [20:07:42] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #13: 04FAILURE in 1 hr 18 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/13/ [20:41:30] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog, 10Patch-For-Review, 10Technical-Debt: Figure out the right way to keep the Android SDK up to date in CI - https://phabricator.wikimedia.org/T158456#3445946 (10Mholloway) 05Open>03Resolved a:03Mholloway CI is up-to-date, passing, and st... [20:43:24] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:52:10] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: MW-1.30.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T167893#3445996 (10thcipriani) [20:53:18] 10Continuous-Integration-Config, 10MinervaNeue: Not possible for a skin to have PHPUnit tests (PHPunit tests for skin do not seem to be run) - https://phabricator.wikimedia.org/T170880#3445999 (10Jdlrobson) [20:55:34] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10MinervaNeue: Not possible for a skin to have PHPUnit tests (PHPunit tests for skin do not seem to be run) - https://phabricator.wikimedia.org/T170880#3446021 (10Jdlrobson) [20:57:52] (03PS1) 10Jdlrobson: test-skins should also run testsuite extensions [integration/config] - 10https://gerrit.wikimedia.org/r/365796 (https://phabricator.wikimedia.org/T170880) [20:58:06] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446038 (10thcipriani) [20:58:28] (03PS2) 10Jdlrobson: test-skins should also run testsuite extensions [integration/config] - 10https://gerrit.wikimedia.org/r/365796 (https://phabricator.wikimedia.org/T170880) [20:58:38] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446053 (10thcipriani) I think all these can be removed Some of these per @hashar @mobrovac @fgiunchedi ``` # Outdated /srv/deployment/parsoid/Parsoid # Integration per Antoine T109926#2096483 /... [20:59:03] 10Scap (Scap3-Adoption-Phase1), 10releng-201516-q4, 10releng-201718-q1, 10Trebuchet: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#3446059 (10thcipriani) [20:59:05] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446058 (10thcipriani) [21:00:06] (03CR) 10Paladox: "This may break other skins tests. Maybe we should only run this for the MinervaNeue skin? Or have a separate test?" [integration/config] - 10https://gerrit.wikimedia.org/r/365796 (https://phabricator.wikimedia.org/T170880) (owner: 10Jdlrobson) [21:00:32] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446085 (10hashar) > # Deprecated -- possibly keep? > /srv/deployment/analytics/kraken I have removed it from CI via T169303. The project has been replaced by `analytics/refinery`. [21:00:44] thcipriani: analytics/kraken can be dropped. I got the confirmation a couple weeks ago [21:00:50] (but forgot about scap :() [21:01:36] I am 100% sure it got confirmed by Nuria :] [21:01:44] hasharAway: yeah, I saw some phab task you were on with that one. [21:02:21] the integration/* I will check them tomorrow :D [21:02:23] afaik they can all be removed [21:02:44] the idea was to eventually use Trebuchet to deploy to the labs instances [21:03:02] but the prod salt master can not chain to a labs salt master... so we went with a git::clone :] [21:03:06] paladox: no other skins have tests [21:03:07] i checked [21:03:18] I mean the extension one [21:03:23] will break it [21:03:38] paladox no other skins have extension dependencies [21:03:43] only MobileFrontend [21:03:45] and Minerva [21:03:45] ok [21:04:01] (03CR) 10Paladox: [C: 031] test-skins should also run testsuite extensions [integration/config] - 10https://gerrit.wikimedia.org/r/365796 (https://phabricator.wikimedia.org/T170880) (owner: 10Jdlrobson) [21:04:12] thcipriani: RainbowSprinkles want to talk enabling Minerva ? [21:04:33] (now wmf9 is everywhere) [21:04:36] (03CR) 10Pmiazga: [C: 04-1] "I'm not sure it fixes the problem. Adding `extensions` testsuite will run extensions tests. but MinervaNeue is located inside skins folder" [integration/config] - 10https://gerrit.wikimedia.org/r/365796 (https://phabricator.wikimedia.org/T170880) (owner: 10Jdlrobson) [21:06:53] (03CR) 10Pmiazga: [C: 031] "ok, but wfLoadSkin() uses ExtensionRegistry::load() same way as wfLoadExtension() does. LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/365796 (https://phabricator.wikimedia.org/T170880) (owner: 10Jdlrobson) [21:07:24] (03CR) 10Jdlrobson: [C: 031] "Can also see that this works by looking at the failure here when using suites=extension:" [integration/config] - 10https://gerrit.wikimedia.org/r/365796 (https://phabricator.wikimedia.org/T170880) (owner: 10Jdlrobson) [21:08:43] I am available [21:09:01] to talk minerva [21:09:27] I don't know what went awry last time tho [21:10:20] thcipriani: so wmf7 is gone right? [21:11:25] jdlrobson: yes indeed https://tools.wmflabs.org/versions/ still monitoring to make sure it's stable but so far so good. [21:13:10] thcipriani: ok cool. So it's got a little messier now we just did those other changes.. but lemme get some patches ready [21:13:56] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:15:48] (03CR) 10Jforrester: [C: 031] "Seems sane?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/365503 (owner: 10Umherirrender) [21:17:06] thcipriani: taking a while to catch up since we've done so many reverts over past few :) [21:19:30] thcipriani: okay, so https://gerrit.wikimedia.org/r/#/c/365861/ can be merged and Minerva enabled on wmf9 [21:19:43] that can be done safely and should ensure the train works [21:20:01] then Minerva will roll out with the train [21:20:16] if we don't do that, the mobile site will break on next train :) [21:21:25] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446229 (10mobrovac) @ssastry @Arlolra can you confirm `/srv/deployment/parsoid/Parsoid` on `tin` is obsolete and can be removed? [21:21:35] well, I like the first option :) [21:23:23] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:25:03] thcipriani: can we do now? :) [21:25:29] we can even avoid the patch https://gerrit.wikimedia.org/r/#/c/365861/ if necessary - it would just mean an untranslated skin on Special:Version [21:25:59] i'm not sure if RainbowSprinkles is around? [21:26:18] are these translations in master? [21:26:19] I am, yes. [21:26:21] Sorry, lunch ran late [21:26:46] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:27:16] thcipriani: yup translation is in master [21:27:37] (https://gerrit.wikimedia.org/r/#/c/365162/) [21:29:49] I'm fine pushing this out now. Does anything else need to happen before train tomorrow? Since this will require a full scap... [21:31:32] thcipriani: i dont think so.. but i defer to RainbowSprinkles . The main thing is we need to enable MinervaNeue on wmf9 [21:31:40] so it shows up as an installed skin [21:31:56] (it will be inert code until the next train) [21:32:10] right now it doesnt show up on Special:Version [21:33:05] alright, I'll go ahead and sync out the l10n update [21:33:13] thcipriani: sweet [21:34:32] uhhh is nodepool suck [21:34:34] 10Scap (Scap3-Adoption-Phase1), 10Trebuchet: Cleanup /srv/deployment - https://phabricator.wikimedia.org/T170881#3446346 (10ssastry) >>! In T170881#3446229, @mobrovac wrote: > @ssastry @Arlolra can you confirm `/srv/deployment/parsoid/Parsoid` on `tin` is obsolete and can be removed? Yes. [21:34:35] * thcipriani looks [21:35:40] eh, maybe just slow [21:36:44] 10Browser-Tests-Infrastructure, 10MinervaNeue: MinervaNeue browser test is flaking - https://phabricator.wikimedia.org/T170890#3446357 (10Jdlrobson) [21:36:45] thcipriani: fwiw I see it churning instances but it's not using its full pool [21:36:53] well, maybe it is but slowly [21:40:00] hrm, it doesn't seem like there's anything "wrong" looking at the logs and the state it thinks its servers are in [21:40:13] but it sure is moving out the backlog kinda slowly [21:41:05] it just tore down and set for build like 8 simultaneiously [21:41:11] so something was eating up a bunch of spots is my best guess [21:41:24] 10Browser-Tests-Infrastructure, 10MinervaNeue, 10Reading-Web-Backlog: MinervaNeue browser test is flaking - https://phabricator.wikimedia.org/T170890#3446357 (10Jdlrobson) [21:41:34] 10Browser-Tests-Infrastructure, 10MinervaNeue, 10Reading-Web-Backlog: MinervaNeue browser test is flaking - https://phabricator.wikimedia.org/T170890#3446357 (10Jdlrobson) p:05Triage>03Normal [21:41:46] 10Browser-Tests-Infrastructure, 10MinervaNeue, 10Reading-Web-Backlog: MinervaNeue browser test is flaking - https://phabricator.wikimedia.org/T170890#3446357 (10Jdlrobson) Need to debug what's going on here before committing to this. [21:41:48] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:43:01] thcipriani, jdlrobson: Yeah plan for today was full scap + config swap for wmf9 (well, all wikis technically) so wmf10 tomorrow rolls out fine [21:43:54] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:44:10] RainbowSprinkles: jdlrobson whenever this merges (which may be a while...) https://gerrit.wikimedia.org/r/#/c/365861/1 I'll scap it out [21:45:57] * RainbowSprinkles nods [21:48:39] RainbowSprinkles: thcipriani sweet [21:48:53] lemme know when i can test [21:54:58] RainbowSprinkles: I don't need to do anything special to ensure scap builds these new messages, right? Am I missing anything? [21:56:43] ah, nevermind, I see it in the extension-list, that's mainly what I was asking about [21:57:29] Yeah, that's all that's needed [21:57:49] thcipriani: Yep, already did that. And l10n was already built, we're just rebuilding for the changes [21:58:56] cool beans [22:26:36] (03CR) 10Legoktm: [C: 032] Add .gitattributes [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/365503 (owner: 10Umherirrender) [22:28:00] (03Merged) 10jenkins-bot: Add .gitattributes [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/365503 (owner: 10Umherirrender) [22:31:17] 10Release-Engineering-Team (Kanban), 10Phabricator, 10User-Zppix: Phabricator max exec time exceeded when going to https://phabricator.wikimedia.org/guides/quickstart/ - https://phabricator.wikimedia.org/T167221#3446606 (10mmodell) 05Open>03Resolved a:03mmodell The guides application is now uninstalled. [22:42:32] 10Release-Engineering-Team, 10MobileFrontend, 10Reading-Web-Backlog, 10RelatedArticles, and 4 others: RelatedPages PHP unit tests are failing during to an issue with MobileFrontend - https://phabricator.wikimedia.org/T170624#3446636 (10Jdlrobson) p:05High>03Normal [22:42:38] 10Release-Engineering-Team, 10MobileFrontend, 10Reading-Web-Backlog, 10RelatedArticles, and 3 others: RelatedPages PHP unit tests are failing during to an issue with MobileFrontend - https://phabricator.wikimedia.org/T170624#3437378 (10Jdlrobson) a:05Jdlrobson>03None [22:43:14] 10Release-Engineering-Team, 10MobileFrontend, 10Reading-Web-Backlog, 10RelatedArticles, and 3 others: RelatedPages PHP unit tests are failing during to an issue with MobileFrontend - https://phabricator.wikimedia.org/T170624#3437378 (10Jdlrobson) [22:43:36] 10Release-Engineering-Team, 10MobileFrontend, 10Reading-Web-Backlog, 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-07-18_(1.30.0-wmf.10)): RelatedPages PHP unit tests are failing during to an issue with MobileFrontend - https://phabricator.wikimedia.org/T170624#3437378 (10Jdlrobson) I've dr... [22:44:33] 10Release-Engineering-Team, 10MobileFrontend, 10Reading-Web-Backlog, 10RelatedArticles, 10MW-1.30-release-notes (WMF-deploy-2017-07-18_(1.30.0-wmf.10)): RelatedPages PHP unit tests are failing during to an issue with MobileFrontend - https://phabricator.wikimedia.org/T170624#3446650 (10Jdlrobson) [23:02:41] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:09:28] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:14:23] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:18:05] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [23:18:39] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:23:23] PROBLEM - Puppet errors on castor is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [23:32:27] 10Release-Engineering-Team, 10Operations, 10Composer: Setup a Composer Repository (Packagist) for MediaWiki Extensions - https://phabricator.wikimedia.org/T170897#3446751 (10Reedy) [23:44:32] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [23:44:34] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:49:26] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [23:49:55] 10Continuous-Integration-Config, 10BlueSpice, 10Patch-For-Review: Enable unit tests on BlueSpice* repos - https://phabricator.wikimedia.org/T130811#3446815 (10Paladox) @Osnard See the change above witch passes the test now :) [23:56:27] 10Continuous-Integration-Infrastructure, 10Wikipedia-Android-App-Backlog: Create a Wikimedia fork of the Jenkins android emulator plugin - https://phabricator.wikimedia.org/T170904#3446832 (10Mholloway) [23:56:48] 10Continuous-Integration-Infrastructure, 10Wikipedia-Android-App-Backlog: Create a Wikimedia fork of the Jenkins android emulator plugin - https://phabricator.wikimedia.org/T170904#3446847 (10Mholloway) [23:58:42] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [23:58:57] 10Continuous-Integration-Infrastructure, 10Wikipedia-Android-App-Backlog: Create a Wikimedia fork of the Jenkins android emulator plugin - https://phabricator.wikimedia.org/T170904#3446848 (10Mholloway)