[00:17:19] 10Continuous-Integration-Infrastructure, 10Core Platform Team Goals (PHP7 (TEC4)), 10Core Platform Team Kanban (Kanban), 10PHP 7.2 support: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10CCicalese_WMF) [00:26:05] 10Deployments, 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Core Platform Team Goals (MCR: Uncategorized), and 3 others: Localization Cache Redo - https://phabricator.wikimedia.org/T78802 (10CCicalese_WMF) [00:26:20] 10Deployments, 10Release-Engineering-Team, 10TechCom-RFC, 10Core Platform Team Goals (MCR: Uncategorized), and 2 others: RFC: Reevaluate LocalisationUpdate extension for WMF - https://phabricator.wikimedia.org/T158360 (10CCicalese_WMF) [00:28:35] 10Continuous-Integration-Infrastructure, 10Core Platform Team Goals (MCR: Uncategorized), 10Core Platform Team Kanban (Later), 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751 (10CCicalese_WMF) [00:32:33] 10Release-Engineering-Team (Watching / External), 10Core Platform Team Goals (MCR: Uncategorized), 10Core Platform Team Kanban (Later): Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a r... - https://phabricator.wikimedia.org/T106388 [00:40:09] maintenance-disconnect-full-disks build 7633 integration-slave-docker-1037 (/srv: 97%): OFFLINE due to disk space [00:42:57] PROBLEM - Free space - all mounts on integration-slave-docker-1037 is CRITICAL: CRITICAL: integration.integration-slave-docker-1037.diskspace._srv.byte_percentfree (<40.00%) [00:50:07] maintenance-disconnect-full-disks build 7635 integration-slave-docker-1037: OFFLINE due to disk space [01:15:07] maintenance-disconnect-full-disks build 7640 integration-slave-docker-1037: OFFLINE due to disk space [01:40:07] maintenance-disconnect-full-disks build 7645 integration-slave-docker-1037: OFFLINE due to disk space [01:47:45] (03PS2) 10Krinkle: [WIP] Add performance-patch job [integration/config] - 10https://gerrit.wikimedia.org/r/459268 (https://phabricator.wikimedia.org/T133646) [02:05:06] maintenance-disconnect-full-disks build 7650 integration-slave-docker-1037: OFFLINE due to disk space [02:20:08] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Kanban), 10PHP 7.2 support: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10CCicalese_WMF) [02:30:06] maintenance-disconnect-full-disks build 7655 integration-slave-docker-1037: OFFLINE due to disk space [02:31:02] 10Continuous-Integration-Infrastructure, 10Core Platform Team, 10Core Platform Team Kanban (Later), 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751 (10CCicalese_WMF) [02:35:09] (03PS1) 10Legoktm: Turn quibble-stretch into a PHP version agnostic image [integration/config] - 10https://gerrit.wikimedia.org/r/463883 [02:36:10] 10Release-Engineering-Team (Watching / External), 10Core Platform Team, 10Core Platform Team Kanban (Later): Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a reference in the text table ... - https://phabricator.wikimedia.org/T106388 [02:37:12] 10Deployments, 10Release-Engineering-Team, 10MediaWiki-extensions-LocalisationUpdate, 10Core Platform Team, and 3 others: Localization Cache Redo - https://phabricator.wikimedia.org/T78802 (10CCicalese_WMF) [02:40:15] (03CR) 10Legoktm: [C: 032] docker: use python-debian to test changelog files [integration/config] - 10https://gerrit.wikimedia.org/r/461090 (owner: 10Hashar) [02:42:17] (03Merged) 10jenkins-bot: docker: use python-debian to test changelog files [integration/config] - 10https://gerrit.wikimedia.org/r/461090 (owner: 10Hashar) [02:42:17] (03CR) 10Legoktm: [C: 032] "Can we do some regex over the dockerfile template to get a list of real dependencies vs. what's declared in the control file? IIRC the old" [integration/config] - 10https://gerrit.wikimedia.org/r/461095 (owner: 10Hashar) [02:42:19] (03CR) 10jerkins-bot: [V: 04-1] docker: test dependencies exist [integration/config] - 10https://gerrit.wikimedia.org/r/461095 (owner: 10Hashar) [02:43:57] (03PS2) 10Legoktm: docker: test dependencies exist [integration/config] - 10https://gerrit.wikimedia.org/r/461095 (owner: 10Hashar) [02:43:57] (03CR) 10Legoktm: [C: 032] "..." [integration/config] - 10https://gerrit.wikimedia.org/r/461095 (owner: 10Hashar) [02:44:09] (03Merged) 10jenkins-bot: docker: test dependencies exist [integration/config] - 10https://gerrit.wikimedia.org/r/461095 (owner: 10Hashar) [02:55:07] maintenance-disconnect-full-disks build 7660 integration-slave-docker-1037: OFFLINE due to disk space [03:09:36] 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10Ejegg) [03:20:07] maintenance-disconnect-full-disks build 7665 integration-slave-docker-1037: OFFLINE due to disk space [03:20:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10Legoktm) [03:24:26] (03PS1) 10Krinkle: Add pipeline for performance/fresnel.git (npm-docker) [integration/config] - 10https://gerrit.wikimedia.org/r/463886 (https://phabricator.wikimedia.org/T133646) [03:24:35] (03CR) 10Krinkle: [C: 032] Add pipeline for performance/fresnel.git (npm-docker) [integration/config] - 10https://gerrit.wikimedia.org/r/463886 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [03:25:59] (03Merged) 10jenkins-bot: Add pipeline for performance/fresnel.git (npm-docker) [integration/config] - 10https://gerrit.wikimedia.org/r/463886 (https://phabricator.wikimedia.org/T133646) (owner: 10Krinkle) [03:26:23] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463886 [03:26:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:45:07] maintenance-disconnect-full-disks build 7670 integration-slave-docker-1037: OFFLINE due to disk space [04:10:07] maintenance-disconnect-full-disks build 7675 integration-slave-docker-1037: OFFLINE due to disk space [04:35:06] maintenance-disconnect-full-disks build 7680 integration-slave-docker-1037: OFFLINE due to disk space [05:00:07] maintenance-disconnect-full-disks build 7685 integration-slave-docker-1037: OFFLINE due to disk space [05:21:19] (03PS3) 10Legoktm: Turn quibble-stretch into a PHP version agnostic image [integration/config] - 10https://gerrit.wikimedia.org/r/463883 [05:22:53] (03PS1) 10Legoktm: Use 0.0.26-1 quibble-stretch docker images [integration/config] - 10https://gerrit.wikimedia.org/r/463890 [05:25:07] maintenance-disconnect-full-disks build 7690 integration-slave-docker-1037: OFFLINE due to disk space [05:30:25] (03CR) 10Legoktm: [C: 032] Turn quibble-stretch into a PHP version agnostic image [integration/config] - 10https://gerrit.wikimedia.org/r/463883 (owner: 10Legoktm) [05:31:49] (03Merged) 10jenkins-bot: Turn quibble-stretch into a PHP version agnostic image [integration/config] - 10https://gerrit.wikimedia.org/r/463883 (owner: 10Legoktm) [05:32:40] !log rebuilding quibble-stretch images https://gerrit.wikimedia.org/r/463883 [05:32:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:50:08] maintenance-disconnect-full-disks build 7695 integration-slave-docker-1037: OFFLINE due to disk space [06:15:06] maintenance-disconnect-full-disks build 7700 integration-slave-docker-1037: OFFLINE due to disk space [06:40:06] maintenance-disconnect-full-disks build 7705 integration-slave-docker-1037: OFFLINE due to disk space [07:05:07] maintenance-disconnect-full-disks build 7710 integration-slave-docker-1037: OFFLINE due to disk space [07:12:40] (03CR) 10Legoktm: [C: 032] Use 0.0.26-1 quibble-stretch docker images [integration/config] - 10https://gerrit.wikimedia.org/r/463890 (owner: 10Legoktm) [07:16:23] (03Merged) 10jenkins-bot: Use 0.0.26-1 quibble-stretch docker images [integration/config] - 10https://gerrit.wikimedia.org/r/463890 (owner: 10Legoktm) [07:30:07] maintenance-disconnect-full-disks build 7715 integration-slave-docker-1037: OFFLINE due to disk space [07:36:16] (03PS1) 10Legoktm: Add quibble-stretch-php71 docker image [integration/config] - 10https://gerrit.wikimedia.org/r/463903 (https://phabricator.wikimedia.org/T204884) [07:36:32] (03CR) 10Legoktm: [C: 032] Add quibble-stretch-php71 docker image [integration/config] - 10https://gerrit.wikimedia.org/r/463903 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [07:37:58] (03Merged) 10jenkins-bot: Add quibble-stretch-php71 docker image [integration/config] - 10https://gerrit.wikimedia.org/r/463903 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [07:38:24] !log building quibble-stretch-php71 docker image [07:38:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:40:12] (03PS1) 10Legoktm: Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 [07:45:17] (03PS2) 10Legoktm: Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 [07:45:42] (03PS3) 10Legoktm: Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 (https://phabricator.wikimedia.org/T204884) [07:47:48] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) As of 2018-10-02: ``` ls -rw-r--r-- 1 dump dump 3.2G Sep 25 21:37 mgwiktionary.gz.tar -rw-r--r-- 1 dump... [07:47:58] (03CR) 10jerkins-bot: [V: 04-1] Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [07:48:05] (03PS4) 10Legoktm: Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 (https://phabricator.wikimedia.org/T204884) [07:49:22] (03PS5) 10Legoktm: Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 (https://phabricator.wikimedia.org/T204884) [07:54:25] (03CR) 10Legoktm: [C: 032] Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [07:55:07] maintenance-disconnect-full-disks build 7720 integration-slave-docker-1037: OFFLINE due to disk space [07:58:09] (03Merged) 10jenkins-bot: Add quibble php71 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463905 (https://phabricator.wikimedia.org/T204884) (owner: 10Legoktm) [07:59:35] !log deployed https://gerrit.wikimedia.org/r/463905 [07:59:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:03:53] (03PS1) 10Legoktm: Remove some no-op branch filters [integration/config] - 10https://gerrit.wikimedia.org/r/463912 [08:07:11] (03CR) 10Legoktm: [C: 04-1] "Wikibase is still using nodepool jobs... :|" [integration/config] - 10https://gerrit.wikimedia.org/r/463912 (owner: 10Legoktm) [08:10:32] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10Petar.petkovic) [08:12:25] 10MediaWiki-Codesniffer, 10MediaWiki-Core-Tests, 10PHP 7.1 support, 10Patch-For-Review: Function mcrypt_list_algorithms() has been deprecated - https://phabricator.wikimedia.org/T201524 (10Legoktm) 05Open>03Resolved a:03Legoktm The problematic sniff has been removed from our ruleset. [08:13:12] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Kanban), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Legoktm) [08:15:23] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Kanban), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Legoktm) MediaWiki core had a clean test run on 7.1. Extension gate ran i... [08:20:07] maintenance-disconnect-full-disks build 7725 integration-slave-docker-1037: OFFLINE due to disk space [08:20:45] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, 10wikitech.wikimedia.org: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) Running to check timing and correctness: ``` # time recover_section.py s3 --database enwikivoyage --host d... [08:24:06] Project beta-scap-eqiad build #223665: 04FAILURE in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223665/ [08:25:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Goal, and 2 others: Add Prometheus exporter to Jenkins instances - https://phabricator.wikimedia.org/T182759 (10fgiunchedi) >>! In T182759#4631716, @dduvall wrote: > As of this morning both Jenkins master have the... [08:39:19] Yippee, build fixed! [08:39:20] Project beta-scap-eqiad build #223666: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223666/ [08:45:06] maintenance-disconnect-full-disks build 7730 integration-slave-docker-1037: OFFLINE due to disk space [09:00:50] 10Beta-Cluster-Infrastructure, 10Wikimedia-Logstash: Logstash in beta doesn't have any logs - https://phabricator.wikimedia.org/T205863 (10fgiunchedi) 05Open>03Resolved a:03fgiunchedi I bounced logstash on deployment-logstash2 and looks like logs are flowing again, logstash-plain.log wasn't being written... [09:10:07] maintenance-disconnect-full-disks build 7735 integration-slave-docker-1037: OFFLINE due to disk space [09:35:07] maintenance-disconnect-full-disks build 7740 integration-slave-docker-1037: OFFLINE due to disk space [10:00:07] maintenance-disconnect-full-disks build 7745 integration-slave-docker-1037: OFFLINE due to disk space [10:24:18] Project beta-scap-eqiad build #223673: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223673/ [10:25:08] maintenance-disconnect-full-disks build 7750 integration-slave-docker-1037: OFFLINE due to disk space [10:38:49] Yippee, build fixed! [10:38:49] Project beta-scap-eqiad build #223674: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/223674/ [10:50:07] maintenance-disconnect-full-disks build 7755 integration-slave-docker-1037: OFFLINE due to disk space [11:15:07] maintenance-disconnect-full-disks build 7760 integration-slave-docker-1037: OFFLINE due to disk space [11:40:07] maintenance-disconnect-full-disks build 7765 integration-slave-docker-1037: OFFLINE due to disk space [12:05:07] maintenance-disconnect-full-disks build 7770 integration-slave-docker-1037: OFFLINE due to disk space [12:30:06] maintenance-disconnect-full-disks build 7775 integration-slave-docker-1037: OFFLINE due to disk space [12:55:09] maintenance-disconnect-full-disks build 7780 integration-slave-docker-1037: OFFLINE due to disk space [13:04:13] 10MediaWiki-Codesniffer, 10Technical-Debt: Warn when a method/class has a @deprecated annotation but no call to wfDeprecated - https://phabricator.wikimedia.org/T205938 (10Anomie) If it warns, won't that prevent soft deprecations entirely? [13:18:08] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Ottomata) > Is the hope that https://gerrit.wikimedia.org/r/#/c/operations/puppe... [13:20:07] maintenance-disconnect-full-disks build 7785 integration-slave-docker-1037: OFFLINE due to disk space [13:20:47] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) [13:28:43] 10Deployments, 10Release-Engineering-Team, 10Core Platform Team Kanban, 10MediaWiki-extensions-LocalisationUpdate, and 3 others: Localization Cache Redo - https://phabricator.wikimedia.org/T78802 (10CCicalese_WMF) [13:45:07] maintenance-disconnect-full-disks build 7790 integration-slave-docker-1037: OFFLINE due to disk space [14:10:08] maintenance-disconnect-full-disks build 7795 integration-slave-docker-1037: OFFLINE due to disk space [14:16:54] 10Continuous-Integration-Infrastructure, 10Wikimedia-General-or-Unknown, 10Core Platform Team, 10Core Platform Team Kanban (Kanban), and 2 others: Deprecate/obsolete $wgWikimediaJenkinsCI - https://phabricator.wikimedia.org/T200650 (10CCicalese_WMF) [14:25:24] 10Release-Engineering-Team (Watching / External), 10Core Platform Team, 10Core Platform Team Kanban (Kanban), 10Epic, 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733 (10CCicalese_WMF) [14:35:08] maintenance-disconnect-full-disks build 7800 integration-slave-docker-1037: OFFLINE due to disk space [14:57:34] hi rel-eng! [14:58:08] this channel seems pretty full of disk space warnings just now [14:58:29] would that have anything to do with the DB problems we've seen in CRM tests? [14:58:47] https://phabricator.wikimedia.org/T205950 [14:59:23] fundraising tech has been force-merging everything for that repo for the past couple weeks [14:59:35] would be great to get back to normal! [15:00:07] maintenance-disconnect-full-disks build 7805 integration-slave-docker-1037: OFFLINE due to disk space [15:01:51] https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm/7367/console says it ran on integration-slave-jessie-1001 [15:02:20] Sep 06 00:55:13 maintenance-disconnect-full-disks build 713 integration-slave-jessie-1001: OFFLINE due to disk space [15:02:41] df -h shows, among other things [15:02:46] Filesystem Size Used Avail Use% Mounted on [15:02:53] /dev/vda3 19G 16G 2.2G 88% / [15:03:04] none 256M 249M 7.9M 97% /var/lib/mysql [15:03:04] /dev/mapper/vd-second--local--disk 21G 14G 5.7G 71% /srv [15:03:11] none 256M 7.4M 249M 3% /srv/home/jenkins-deploy/tmpfs [15:03:53] so no wonder mysql stuff is failing [15:05:26] mysql appears to have two DBs (excluding mysql, information_schema and performance_schema): [15:05:26] | civicrm_jenkins_wikimedia_fundraising_civicrm_7351_2 | [15:05:26] | civicrm_jenkins_wikimedia_fundraising_civicrm_7351_3 | [15:05:47] the first of which has no tables [15:12:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10Krenair) https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm/7367/console says it ran on integration-slave-jessie-1001 Sep 06 00:55:13 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10Krenair) I found some fundraising tables lying around on some integration hosts but it doesn't look like they're responsible being only 20MB each: relevant stuf... [15:15:24] <_joe_> thcipriani: around? I created a patch to scap for automatically choose the right canaries. [15:15:37] <_joe_> s/choose/choosing/ [15:15:38] _joe_: yep [15:15:57] <_joe_> it will need me and alex to merge a puppet patch before it can even hope to work :P [15:16:27] fair warning :) [15:17:10] <_joe_> that is, adding the "canary" service in etcd, and also adding the new autogenerated dsh files :) [15:17:11] this would be easier if the the active deployment server was in the active datacenter. Scap's config was built with that in mind, mostly. [15:17:39] <_joe_> we don't want that to be a tight coupling [15:17:52] <_joe_> so adding a bit of complexity to scap is worth it IMHO [15:18:47] yeah, I noticed that's not the desired behavior any more :) I'll take a look at the patch. [15:20:50] <_joe_> I don't think tight coupling was ever desired [15:21:10] <_joe_> but it was an historical truth :) [15:21:58] 10Project-Admins: create a new project for WMTechConf - https://phabricator.wikimedia.org/T205983 (10debt) [15:22:17] hysterical raisins [15:25:06] maintenance-disconnect-full-disks build 7810 integration-slave-docker-1037: OFFLINE due to disk space [15:41:50] thanks for investigating, Krenair! What's the right way to purge the DBs? I've been able to delete full jenkins workspaces on integration hosts in the past. [15:45:24] ejegg, I have no idea [15:45:48] ah, ok. Thanks for pinpointing the problem [15:46:03] I tend not to touch the integration project [15:46:08] or any CI stuff really [15:47:19] I could have found entirely the wrong thing [15:47:40] huh, I only see the one civi db, of 20mb, on that integration host [15:47:46] yeah [15:47:57] hopefully I was more helpful than that but I won't really know until someone who is actually familiar with our CI takes a look [15:48:17] right on, thanks for jumping in [15:50:06] maintenance-disconnect-full-disks build 7815 integration-slave-docker-1037: OFFLINE due to disk space [15:50:27] especially those docker hosts [15:50:31] no idea how that all works [16:07:36] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10Krenair) [16:13:07] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10CCicalese_WMF) @Krenair Thanks for tagging! I meant to do that . . . [16:15:07] maintenance-disconnect-full-disks build 7820 integration-slave-docker-1037: OFFLINE due to disk space [16:20:48] _joe_: added some review on the scap patch, mostly nits, but I do have a worry about depool working now [16:20:59] !log investigating integration-slave-docker-1037 [16:21:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:21:10] <_joe_> thcipriani: ack, will look tomorrow [16:36:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10Ejegg) Looks like the ibdata1 file is taking up almost all the space: root@integration-slave-jessie-1003:/var/lib/mysql# du -sh * 1.3M civicrm_jenkins_wikimedia... [16:41:28] we have some reverts to make on the mw branch that has just been cut for the WikibaseLexeme extension FYI [16:41:32] patches incomming [16:41:45] marxarelli: ^^ [16:41:56] tarrow: jakob_WMDE ^^ :) [16:44:27] revert away. i'll be sure to incorporate them before i deploy [16:45:06] marxarelli: cool, once they are ready I'll just +2 them on the branch ready? :) [16:45:19] yes! [16:45:23] great [16:45:24] ty [16:45:28] I'll give you a ping once done too [16:45:33] !log bring integration-slave-docker-1037 back online after rm -rf /srv/jenkins-workspace/workspace/* [16:45:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:51:36] marxarelli: this looked like stuff you might be interested in -- https://opensource.com/article/18/9/quiet-log-noise-python-and-machine-learning [16:57:53] RECOVERY - Free space - all mounts on integration-slave-docker-1037 is OK: OK: All targets OK [16:59:03] (03CR) 10Krinkle: [C: 031] "Very nice :) – The wiping through a docker command surprises me somewhat though, I thought Jenkins has a built-in workspace option as well" [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [17:03:52] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) The full import process took around 7-8 hours (although it was quite inefficient to re-import just one database at a time... [17:16:42] 10Release-Engineering-Team (Watching / External), 10DBA, 10Operations, 10cloud-services-team, and 2 others: Move some wikis to s5 - https://phabricator.wikimedia.org/T184805 (10jcrespo) Command executed, for posterity: ``` root@db1110.eqiad.wmnet[(none)]> CHANGE MASTER 's3' TO ... ... [17:23:40] 10Scap, 10Operations, 10Product-Analytics, 10SRE-Access-Requests, and 2 others: Add Mathew.onipe(onimisionipe) to deployment group - https://phabricator.wikimedia.org/T205981 (10herron) p:05Triage>03Normal [17:24:51] (03PS1) 10Legoktm: Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 [17:27:31] (03CR) 10jerkins-bot: [V: 04-1] Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 (owner: 10Legoktm) [17:28:49] (03PS2) 10Legoktm: Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 [17:30:12] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10ovasileva) [17:30:56] (03CR) 10jerkins-bot: [V: 04-1] Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 (owner: 10Legoktm) [17:31:34] 10Project-Admins: create a new project for WMTechConf - https://phabricator.wikimedia.org/T205983 (10Rfarrand) We should have one here, right? https://phabricator.wikimedia.org/project/view/3322/ [17:35:29] marxarelli: I +2ed the 4 patches on https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions/WikibaseLexeme+branch:wmf/1.32.0-wmf.24+topic:revert-forms-statements-api [17:35:30] 10Project-Admins: create a new project for WMTechConf - https://phabricator.wikimedia.org/T205983 (10debt) 05Open>03Invalid Nevermind!! I didn't realize we already had a project setup. [17:35:42] i have to leave in 25 mins and they might not all get merged by jenkins before them [17:35:57] also some random CI might fail, but please do check that they are in before rolling the train :) [17:39:15] (03PS3) 10Legoktm: Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 [17:41:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Database busted for CiviCRM tests? - https://phabricator.wikimedia.org/T205950 (10thcipriani) Since there were no databases on the affected boxes that had any schema, I've backed up the ibdata1 file and restarted mysql as there were no hung t... [18:00:23] hey folks .. something change in beta labs config ... parsoid config requests to the mediawiki api are getting an error .. Protocol "http:" not supported. Expected "https:" [18:01:01] RECOVERY - Free space - all mounts on integration-slave-docker-1033 is OK: OK: All targets OK [18:02:07] subbu, I don't have time to dig into this right now, maybe open a task [18:02:17] k [18:02:25] I'd check to see whether your config for beta is truly equivalent to your config for prod [18:02:56] there was a time when beta didn't support HTTPS so parsoid may have had a good reason to be different in that respect [18:03:00] reminder: we (releng) don't know of every change in beta so self investigation is needed :) [18:05:25] fair enough ... we have in the past and will :), but also checking just in case others have a clue or might have encountered this. [18:06:22] I believe it’s because prod is using https [18:06:24] For parsoid [18:06:33] Same problem as wikitech I think [18:06:36] subbu: ^^ [18:06:55] ya .. but parsoid has been working in beta till earlier this morning. [18:07:37] subbu: yeah, sorry, I know you know, I'm also just in a meeting right now :P [18:07:42] no worries :) [18:11:46] (03PS4) 10Legoktm: Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 [18:11:53] (03CR) 10Legoktm: [C: 032] Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 (owner: 10Legoktm) [18:14:55] (03Merged) 10jenkins-bot: Run quibble PHP 7.1 experimental for everything [integration/config] - 10https://gerrit.wikimedia.org/r/464000 (owner: 10Legoktm) [18:15:47] !log deploying https://gerrit.wikimedia.org/r/464000 [18:15:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:20:18] PROBLEM - Free space - all mounts on deployment-deploy01 is CRITICAL: CRITICAL: deployment-prep.deployment-deploy01.diskspace.root.byte_percentfree (<11.11%) [18:22:16] !log deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* [18:22:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:35:18] RECOVERY - Free space - all mounts on deployment-deploy01 is OK: OK: All targets OK [18:40:41] 10Beta-Cluster-Infrastructure, 10Parsoid, 10Services (watching): Beta Cluster: Parsoid config request failures from the MediaWiki API - https://phabricator.wikimedia.org/T206003 (10Pchelolo) The problem is that MW API is configured as 'http://', but for some reason `request` uses the `https` Agent - thus the... [18:52:57] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10Aklapper) Is there a reason why #Core_Platform_Team_Kanban was not made a [[ https://www.mediawiki.org/wiki/Phabricator/Project_management#Parent_Projects,_Subprojects_and_Milestones | subproject ]] of #Core_Plat... [18:58:59] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10CCicalese_WMF) We need each task to be in both projects in a different workboard column in each, since one project represents the current status of the task and the other represents the CPT goal or program it is... [19:00:21] 10Project-Admins: create a new project for WMTechConf - https://phabricator.wikimedia.org/T205983 (10Aklapper) https://phabricator.wikimedia.org/project/query/advanced/ allows searching for existing projects. [19:00:31] 10Project-Admins: create a new project for WMTechConf - https://phabricator.wikimedia.org/T205983 (10Aklapper) [19:00:34] 10Project-Admins, 10Developer-Advocacy (Apr-Jun-2018): Create 4 event projects for Wikimania Hackathon 2018 and 2018 Wikimedia Technical Conference - https://phabricator.wikimedia.org/T191372 (10Aklapper) [19:02:43] 10Continuous-Integration-Infrastructure, 10Core Platform Team (PHP7 (TEC4)), 10Core Platform Team Kanban (Current), 10PHP 7.2 support, 10Patch-For-Review: Run MediaWiki tests on PHP 7.2 - https://phabricator.wikimedia.org/T204884 (10Legoktm) [19:21:16] addshore: great. ty! [19:39:29] marxarelli: looks like jenkins merged them all just fine :) [19:45:05] addshore: yep! and i can confirm the merged changes were pulled down before the sync [19:45:47] bd808: oh, thanks. looks interesting [19:46:48] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10Aklapper) 05Open>03Resolved a:03Aklapper Ah, makes sense! (Sorry, I obviously did not read closely enough.) Thanks. Created H299 and H300. [19:56:27] (03CR) 10Dduvall: "> Patch Set 1: Code-Review+1" [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [20:01:55] twentyafterfour: Is https://phabricator.wikimedia.org/D1114 live in beta? If so, I'll run some tests to try and (not) break beta :) [20:04:25] 10Beta-Cluster-Infrastructure, 10Analytics, 10EventBus, 10MediaWiki-JobQueue: GlobalRename stuck again at Beta - https://phabricator.wikimedia.org/T194376 (10Pchelolo) 05Open>03Resolved a:03Pchelolo I believe that's not an issue any more? [20:13:01] Krinkle: it is live in beta, tests appreciated :) [20:40:53] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Quibble, 10User-zeljkofilipin: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests - https://phabricator.wikimedia.org/T199116 (10Niedzielski) > Running multiple npm install might slow Qu... [20:41:13] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T191070 (10Krinkle) [21:03:30] (03PS1) 10Thcipriani: Add mediawiki/servies/zuul to the CD Pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/464048 (https://phabricator.wikimedia.org/T201611) [21:04:12] whoops [21:04:29] zuul -> zotero [21:04:50] (03PS2) 10Thcipriani: Add mediawiki/services/zotero to the CD Pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/464048 (https://phabricator.wikimedia.org/T201611) [21:06:13] inception [21:14:10] just wait until blubberoid is a thing [21:15:20] I'm excited [21:17:45] (03Abandoned) 10EBernhardson: Don't run search-mjolnir-tox-docker concurrently [integration/config] - 10https://gerrit.wikimedia.org/r/451999 (https://phabricator.wikimedia.org/T201479) (owner: 10EBernhardson) [21:19:03] 10Continuous-Integration-Config, 10Patch-For-Review: Submitting multiple mjolnir patches receive -1 with OSError: [Errno 12] Cannot allocate memory - https://phabricator.wikimedia.org/T201479 (10EBernhardson) 05Open>03Resolved a:03EBernhardson mjolnir tox jobs look to be running on m4executor, not sure w... [21:29:03] 10Phabricator: Create Core Platform Team Herald rules - https://phabricator.wikimedia.org/T205973 (10CCicalese_WMF) Thank you! [21:30:23] (03CR) 10Dduvall: "The `workspace-cleanup` publisher appears to first move the workspace directory to a temporary location and then uses `hudson.FilePath.del" [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [21:30:34] (03CR) 10Thcipriani: [C: 032] Add mediawiki/services/zotero to the CD Pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/464048 (https://phabricator.wikimedia.org/T201611) (owner: 10Thcipriani) [21:32:43] (03Merged) 10jenkins-bot: Add mediawiki/services/zotero to the CD Pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/464048 (https://phabricator.wikimedia.org/T201611) (owner: 10Thcipriani) [21:33:26] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:34:44] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/464048/ [21:34:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:38:47] 10Continuous-Integration-Infrastructure, 10OOUI: Host OOUI PHP demo (and all others?) on a PHP 7 capable server, because it needs PHP 7 - https://phabricator.wikimedia.org/T206046 (10Jdforrester-WMF) [21:42:51] (03CR) 10Dduvall: "Specifying an `include: *` pattern for `workspace-cleanup` does change the behavior; a rename isn't performed first and the delete method " [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [21:53:16] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [22:00:55] 10Gerrit: Gitiles project landing pages should have an anonymous clone URL - https://phabricator.wikimedia.org/T206049 (10Tgr) [22:04:25] PROBLEM - SSH on integration-slave-docker-1021 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:31] (03CR) 10Dduvall: "> I'll try the `docker-wipe-workspace` publisher... and now I _really_ hope it works. :/" [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [22:20:00] (03PS2) 10Dduvall: Wipe entire workspace after builds of Docker based jobs [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) [22:23:41] (03CR) 10Dduvall: [C: 031] "Refactored and verified against at least one Quibble job." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/463848 (https://phabricator.wikimedia.org/T205902) (owner: 10Dduvall) [22:24:23] RECOVERY - SSH on integration-slave-docker-1021 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u7 (protocol 2.0) [23:32:21] 10Beta-Cluster-Infrastructure, 10Analytics-Kanban, 10Operations, 10Patch-For-Review, and 2 others: Prometheus resources in deployment-prep to create grafana graphs of EventLogging - https://phabricator.wikimedia.org/T204088 (10Jdlrobson) @Ottomata still not seeing them.. does that mean https://gerrit.wikim... [23:49:39] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T191070 (10Krinkle)