[00:00:43] RECOVERY - Puppet staleness on deployment-puppetmaster is OK Less than 1.00% above the threshold [3600.0] [00:18:19] PROBLEM - Puppet failure on deployment-apertium01 is CRITICAL 22.22% of data above the critical threshold [0.0] [00:20:13] PROBLEM - Puppet failure on deployment-zookeeper01 is CRITICAL 33.33% of data above the critical threshold [0.0] [00:20:13] PROBLEM - Puppet failure on deployment-parsoid05 is CRITICAL 22.22% of data above the critical threshold [0.0] [00:20:29] PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL 40.00% of data above the critical threshold [0.0] [00:22:04] PROBLEM - Puppet failure on deployment-urldownloader is CRITICAL 55.56% of data above the critical threshold [0.0] [00:22:25] PROBLEM - Puppet failure on deployment-zotero01 is CRITICAL 70.00% of data above the critical threshold [0.0] [02:58:06] Project browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce build #697: FAILURE in 3 min 5 sec: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-mediawiki.org-linux-firefox-sauce/697/ [03:19:16] Yippee, build fixed! [03:19:17] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #797: FIXED in 37 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-firefox-sauce/797/ [03:26:43] 10Beta-Cluster, 10Fundraising Tech Backlog, 6Labs, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574042 (10Krenair) [03:31:46] 10Beta-Cluster, 10Fundraising Tech Backlog, 6Labs, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574045 (10Krenair) I looked at https://wikitech.wikimedia.org/wiki/Special:NovaAddress for depl... [03:35:20] 10Beta-Cluster, 10Fundraising Tech Backlog, 6Labs, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574047 (10Krenair) Since this is urgent, I used "Add host name" on deployment-cache-mobile04, a... [03:37:43] 10Beta-Cluster, 10Fundraising Tech Backlog, 6Labs, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574049 (10awight) [03:52:59] 10Beta-Cluster, 10Fundraising Tech Backlog, 6Labs, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574052 (10Krenair) ```alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (master)$ grep... [04:05:30] 10Beta-Cluster, 10Fundraising Tech Backlog, 6Labs, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574061 (10awight) http://meta.m.wikimedia.beta.wmflabs.org/wiki/Main_Page Hey, you did it! [04:07:19] 10Beta-Cluster, 10Fundraising Tech Backlog, 6Labs, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574062 (10awight) Woohoo, removing the blocked task link so you can finish cleaning up the docs... [04:35:56] 10Beta-Cluster, 10Fundraising Tech Backlog, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574076 (10greg) [04:37:23] Krenair: thanks for responding quickly to that wiki creation in beta cluster [04:41:54] 10Beta-Cluster, 10Fundraising Tech Backlog, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574088 (10AndyRussG) Thanks, all!! :) [04:54:17] 10Continuous-Integration-Infrastructure, 7Composer: Come up with non sucky solution for running "composer test" on repos that have vendor/ checked in - https://phabricator.wikimedia.org/T92605#1574097 (10Tgr) Just put dev dependencies in gitignore? One could probably even write a composer plugin that does that... [05:07:50] 10Beta-Cluster, 10Fundraising Tech Backlog, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574121 (10awight) 5Open>3Resolved a:3awight Confirmed working. Blindingly fast! [05:19:18] 10Beta-Cluster, 10Fundraising Tech Backlog, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574135 (10awight) 5Resolved>3Open Sorry, reading Krenair's comments I see that there might be more w... [05:37:10] Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #522: FAILURE in 35 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/522/ [06:38:24] RECOVERY - Free space - all mounts on deployment-bastion is OK All targets OK [07:06:31] 10Browser-Tests: Investigate distribution of browser test run time - https://phabricator.wikimedia.org/T104396#1574449 (10hashar) I am speculating we have timing in Raita? [07:11:47] !log puppet fails on most beta cluster instances :-( [07:11:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [07:16:29] !log started puppetmaster on deployment-puppetmaster [07:16:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [07:24:48] !log resetted beta cluster puppet master to origin/production . We have lost any cherry pick that might have existed [07:24:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [07:27:27] RECOVERY - Puppet failure on deployment-zotero01 is OK Less than 1.00% above the threshold [0.0] [07:29:47] RECOVERY - Puppet failure on deployment-cxserver03 is OK Less than 1.00% above the threshold [0.0] [07:30:15] RECOVERY - Puppet failure on deployment-parsoid05 is OK Less than 1.00% above the threshold [0.0] [07:31:28] 10Beta-Cluster: Beta cluster puppetmaster lost cherry picked patches due to disk corruption - https://phabricator.wikimedia.org/T110303#1574469 (10hashar) 3NEW [07:31:54] RECOVERY - Puppet failure on deployment-mediawiki03 is OK Less than 1.00% above the threshold [0.0] [07:32:09] RECOVERY - Puppet failure on deployment-mediawiki02 is OK Less than 1.00% above the threshold [0.0] [07:36:04] RECOVERY - Puppet failure on deployment-kafka02 is OK Less than 1.00% above the threshold [0.0] [07:36:42] RECOVERY - Puppet failure on deployment-stream is OK Less than 1.00% above the threshold [0.0] [07:36:54] RECOVERY - Puppet failure on deployment-pdf01 is OK Less than 1.00% above the threshold [0.0] [07:36:56] RECOVERY - Puppet failure on deployment-restbase02 is OK Less than 1.00% above the threshold [0.0] [07:37:08] RECOVERY - Puppet failure on deployment-db2 is OK Less than 1.00% above the threshold [0.0] [07:37:18] RECOVERY - Puppet failure on deployment-sca01 is OK Less than 1.00% above the threshold [0.0] [07:37:38] RECOVERY - Puppet failure on deployment-pdf02 is OK Less than 1.00% above the threshold [0.0] [07:38:00] 10Beta-Cluster: Beta cluster puppetmaster lost cherry picked patches due to disk corruption - https://phabricator.wikimedia.org/T110303#1574481 (10hashar) The only lost patch is apparently https://gerrit.wikimedia.org/r/#/c/197655/ //Move web::sites to web::prod_sites; begin unification in new class// by @demon... [07:38:11] 10Beta-Cluster: Beta cluster puppetmaster lost cherry picked patches due to disk corruption - https://phabricator.wikimedia.org/T110303#1574483 (10hashar) 5Open>3Resolved a:3hashar Seems all good now [07:38:43] !log enabling puppet agent on deployment-puppetmaster. It is disable with no reason given [07:38:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [07:39:02] RECOVERY - Puppet failure on deployment-videoscaler01 is OK Less than 1.00% above the threshold [0.0] [07:39:13] !log puppet is back in action on beta cluster [07:39:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [07:40:07] RECOVERY - Puppet failure on deployment-sca02 is OK Less than 1.00% above the threshold [0.0] [07:40:57] RECOVERY - Puppet failure on deployment-memc04 is OK Less than 1.00% above the threshold [0.0] [07:41:20] RECOVERY - Puppet failure on deployment-memc03 is OK Less than 1.00% above the threshold [0.0] [07:41:30] RECOVERY - Puppet failure on deployment-memc02 is OK Less than 1.00% above the threshold [0.0] [07:42:02] RECOVERY - Puppet failure on deployment-salt is OK Less than 1.00% above the threshold [0.0] [07:42:16] RECOVERY - Puppet failure on deployment-elastic07 is OK Less than 1.00% above the threshold [0.0] [07:42:40] RECOVERY - Puppet failure on deployment-fluorine is OK Less than 1.00% above the threshold [0.0] [07:45:04] RECOVERY - Puppet failure on deployment-redis01 is OK Less than 1.00% above the threshold [0.0] [07:45:48] RECOVERY - Puppet failure on deployment-eventlogging02 is OK Less than 1.00% above the threshold [0.0] [07:46:50] RECOVERY - Puppet failure on deployment-sentry2 is OK Less than 1.00% above the threshold [0.0] [07:47:04] RECOVERY - Puppet failure on deployment-elastic05 is OK Less than 1.00% above the threshold [0.0] [07:48:31] RECOVERY - Puppet failure on deployment-jobrunner01 is OK Less than 1.00% above the threshold [0.0] [07:51:27] RECOVERY - Puppet failure on deployment-upload is OK Less than 1.00% above the threshold [0.0] [07:51:59] RECOVERY - Puppet failure on deployment-puppetmaster is OK Less than 1.00% above the threshold [0.0] [07:52:15] RECOVERY - Puppet failure on deployment-elastic08 is OK Less than 1.00% above the threshold [0.0] [07:52:59] RECOVERY - Puppet failure on deployment-redis02 is OK Less than 1.00% above the threshold [0.0] [07:53:03] RECOVERY - Puppet failure on deployment-mediawiki01 is OK Less than 1.00% above the threshold [0.0] [07:53:07] RECOVERY - Puppet failure on deployment-db1 is OK Less than 1.00% above the threshold [0.0] [07:54:31] RECOVERY - Puppet failure on deployment-mathoid is OK Less than 1.00% above the threshold [0.0] [07:55:17] RECOVERY - Puppet failure on deployment-bastion is OK Less than 1.00% above the threshold [0.0] [07:55:28] RECOVERY - Puppet failure on deployment-elastic06 is OK Less than 1.00% above the threshold [0.0] [07:55:30] RECOVERY - Puppet failure on deployment-logstash2 is OK Less than 1.00% above the threshold [0.0] [07:57:06] RECOVERY - Puppet failure on deployment-urldownloader is OK Less than 1.00% above the threshold [0.0] [07:58:19] RECOVERY - Puppet failure on deployment-apertium01 is OK Less than 1.00% above the threshold [0.0] [08:00:07] RECOVERY - Puppet failure on deployment-zookeeper01 is OK Less than 1.00% above the threshold [0.0] [08:03:55] 10Beta-Cluster, 10MediaWiki-extensions-PoolCounter: Install and configure pool counter on beta cluster - https://phabricator.wikimedia.org/T72940#1574502 (10hashar) [08:09:16] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1574509 (10hashar) [08:53:04] (03Abandoned) 10Hashar: Add Android emulator wrapper [integration/jenkins-job-builder] - 10https://gerrit.wikimedia.org/r/231722 (https://phabricator.wikimedia.org/T107336) (owner: 10Niedzielski) [08:54:43] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: JJB support for Android Emulator plugin - https://phabricator.wikimedia.org/T110307#1574581 (10hashar) 3NEW a:3hashar [08:55:35] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: Create jenkins slave instance dedicated to Android runs - https://phabricator.wikimedia.org/T107336#1574600 (10hashar) I have forked JJB support to sub task T110307 [08:57:58] 6Release-Engineering: Organize browsertests/Selenium training - https://phabricator.wikimedia.org/T100170#1574604 (10zeljkofilipin) 5Open>3Resolved We had training sessions at #wikimedia-hackathon-2015, but not at #wikimania-hackathon-2015. As far as I know, there are no plans for further training this year.... [09:14:04] (03PS1) 10Hashar: Create apps-android-wikipedia-emulator [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) [09:14:47] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: JJB support for Android Emulator plugin - https://phabricator.wikimedia.org/T110307#1574619 (10hashar) Created the job https://integration.wikimedia.org/ci/job/apps-... [09:15:10] (03CR) 10jenkins-bot: [V: 04-1] Create apps-android-wikipedia-emulator [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [09:16:11] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: JJB support for Android Emulator plugin - https://phabricator.wikimedia.org/T110307#1574625 (10hashar) [09:16:13] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: Create jenkins slave instance dedicated to Android runs - https://phabricator.wikimedia.org/T107336#1574624 (10hashar) [09:16:23] 10Continuous-Integration-Config, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: Jenkins should run tests for the Wikipedia app before merge - https://phabricator.wikimedia.org/T62720#1574626 (10hashar) [09:16:24] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: JJB support for Android Emulator plugin - https://phabricator.wikimedia.org/T110307#1574581 (10hashar) [09:28:46] (03PS2) 10Hashar: Create apps-android-wikipedia-emulator [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) [09:29:37] (03CR) 10jenkins-bot: [V: 04-1] Create apps-android-wikipedia-emulator [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [09:34:33] (03PS1) 10Zfilipin: Move browsertests-Wikidata-WikidataTests-linux-chrome-sauce job to JJB [integration/config] - 10https://gerrit.wikimedia.org/r/233925 (https://phabricator.wikimedia.org/T106839) [09:39:13] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 7Technical-Debt: move browsertests-Wikidata-WikidataTests-linux-chrome-sauce to jjb - https://phabricator.wikimedia.org/T106839#1574654 (10zeljkofilipin) From https://integration.wikimedia.org/ci/job/integration-jjb-config-diff/3587/consoleFul... [09:39:38] (03PS2) 10Zfilipin: Move browsertests-Wikidata-WikidataTests-linux-chrome-sauce job to JJB [integration/config] - 10https://gerrit.wikimedia.org/r/233925 (https://phabricator.wikimedia.org/T106839) [09:39:47] (03PS3) 10Hashar: Create apps-android-wikipedia-emulator [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) [09:39:59] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 7Technical-Debt: move browsertests-Wikidata-WikidataTests-linux-chrome-sauce to jjb - https://phabricator.wikimedia.org/T106839#1574656 (10zeljkofilipin) a:3zeljkofilipin [09:41:03] (03CR) 10jenkins-bot: [V: 04-1] Create apps-android-wikipedia-emulator [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [09:41:15] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: JJB support for Android Emulator plugin - https://phabricator.wikimedia.org/T110307#1574657 (10hashar) XML is legit, managed to generate the manually created job and... [09:41:36] (03CR) 10WMDE-Fisch: [C: 031] Move browsertests-Wikidata-WikidataTests-linux-chrome-sauce job to JJB [integration/config] - 10https://gerrit.wikimedia.org/r/233925 (https://phabricator.wikimedia.org/T106839) (owner: 10Zfilipin) [09:43:51] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 7Technical-Debt: move browsertests-Wikidata-WikidataTests-linux-chrome-sauce to jjb - https://phabricator.wikimedia.org/T106839#1574659 (10zeljkofilipin) I have refreshed the job with the proposed patch, please review the changes and let me kn... [09:51:21] Yippee, build fixed! [09:51:22] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #539: FIXED in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/539/ [09:51:26] yeah!!!!!! [09:53:52] 10Beta-Cluster, 10Fundraising Tech Backlog, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1574672 (10jayvdb) [09:57:45] !log Bumping our JJB mirror a3aef64..f01628c Required for the Android Emulator plugin support ( https://phabricator.wikimedia.org/T110307 ) [09:57:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [09:58:32] 10Continuous-Integration-Config, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: Jenkins should run tests for the Wikipedia app before merge - https://phabricator.wikimedia.org/T62720#1574701 (10hashar) [09:58:34] 10Continuous-Integration-Config, 6Release-Engineering, 3Mobile-App-Sprint-64-Android-Gadolinium, 5Patch-For-Review, 3Wikipedia-Android-App: JJB support for Android Emulator plugin - https://phabricator.wikimedia.org/T110307#1574698 (10hashar) 5Open>3Resolved Bumped JJB a3aef64..f01628c The job defin... [10:00:26] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [10:08:53] (03CR) 10Hashar: [C: 031] "The JJB diff failure was resolved by bumping our JJB snapshot to latest upstream which includes support for the android-emulator wrapper." [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [10:09:44] 10Beta-Cluster, 10Continuous-Integration-Infrastructure, 10pywikibot-core, 5Patch-For-Review, 7Pywikibot-tests: Run pywikibot test suite regularly on beta cluster as part of MediaWiki/Wikimedia CI - https://phabricator.wikimedia.org/T100903#1574711 (10jayvdb) >>! In T100903#1566414, @hashar wrote: > http... [10:20:41] 10Continuous-Integration-Infrastructure: MediaWiki phpunit job failing with "Can't connect to local MySQL server" - https://phabricator.wikimedia.org/T109704#1574721 (10hashar) 5Open>3Resolved a:3hashar Sounds likely. Maybe the instant rebooted and deadlocked trying to spanw ElasticSearch. Seems that occu... [10:22:11] 6Release-Engineering: Add all of the KPIs to a single dashboard - https://phabricator.wikimedia.org/T108744#1574725 (10hashar) If we have all KPI metrics in Graphite, we could create such dashboard on https://grafana.wikimedia.org/ :-} [10:33:07] 10Continuous-Integration-Infrastructure, 6Release-Engineering, 7Jenkins: Implement "Jenkins uptime" KPI - https://phabricator.wikimedia.org/T108769#1574749 (10hashar) p:5Triage>3Low Jenkins is only a piece of the tool chain and it is barely down (though I have no metric to backup that claim). The most t... [10:40:47] 6Release-Engineering: Create a Releng dashboard on Grafana - https://phabricator.wikimedia.org/T110313#1574777 (10hashar) 3NEW a:3hashar [10:49:07] 6Release-Engineering: Create a Releng dashboard on Grafana - https://phabricator.wikimedia.org/T110313#1574798 (10hashar) I have edited the landing page at https://grafana.wikimedia.org/ to add a 'RelEng' text box that points to https://grafana.wikimedia.org/#/dashboard/db/releng-main-page We could list all our... [11:13:33] 6Release-Engineering: Create a Releng dashboard on Grafana - https://phabricator.wikimedia.org/T110313#1574866 (10hashar) [12:34:01] 6Release-Engineering, 6Commons, 10MediaWiki-File-management, 10MediaWiki-Tarball-Backports, and 7 others: InstantCommons broken by switch to HTTPS - https://phabricator.wikimedia.org/T102566#1575084 (10BBlack) Pinged both of those tasks. I'm pretty much out of patience with waiting for PHP to suddenly bec... [12:48:48] (03CR) 10JanZerebecki: [C: 032] Move browsertests-Wikidata-WikidataTests-linux-chrome-sauce job to JJB [integration/config] - 10https://gerrit.wikimedia.org/r/233925 (https://phabricator.wikimedia.org/T106839) (owner: 10Zfilipin) [12:50:38] (03Merged) 10jenkins-bot: Move browsertests-Wikidata-WikidataTests-linux-chrome-sauce job to JJB [integration/config] - 10https://gerrit.wikimedia.org/r/233925 (https://phabricator.wikimedia.org/T106839) (owner: 10Zfilipin) [12:54:27] 10Continuous-Integration-Infrastructure, 7Technical-Debt: Delete old jobs not (or no longer) managed by JJB - https://phabricator.wikimedia.org/T91410#1575121 (10JanZerebecki) [12:54:29] 10Continuous-Integration-Config, 10Wikidata, 5Patch-For-Review, 7Technical-Debt: move browsertests-Wikidata-WikidataTests-linux-chrome-sauce to jjb - https://phabricator.wikimedia.org/T106839#1575119 (10JanZerebecki) 5Open>3Resolved Thx. [12:59:10] Yippee, build fixed! [12:59:11] Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #765: FIXED in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/765/ [13:05:33] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1575146 (10hashar) Zeljko rebuild integration-slave-precise-1014 :-} [13:06:17] !sal [13:06:17] https://tools.wmflabs.org/sal/releng [13:28:51] 10Continuous-Integration-Infrastructure, 5Patch-For-Review: Create CI slaves using Debian Jessie (tracking) - https://phabricator.wikimedia.org/T94836#1575200 (10hashar) [13:28:53] 10Continuous-Integration-Infrastructure, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Build Debian package ruby-jsduck for Jessie - https://phabricator.wikimedia.org/T95008#1575198 (10hashar) 5Resolved>3declined We ended up NOT building Debian package ruby-jsduck for Jessie. So adjust status... [13:35:18] 10Continuous-Integration-Infrastructure, 6operations: Update RDiscount gem/package on jenkins build servers (UbuntuTrusty) - https://phabricator.wikimedia.org/T109005#1575219 (10hashar) We wanted to get a Debian package for JSDuck on Jessie (T95008). The packaging works involved ends up being rather painful s... [13:36:22] 10Continuous-Integration-Infrastructure, 6operations: Update RDiscount gem/package on jenkins build servers (UbuntuTrusty) - https://phabricator.wikimedia.org/T109005#1575225 (10hashar) @cscott mind if we change this task to #continuous-integration-config and retitle it to switch CI from jsduck deb package to... [13:37:34] 10Beta-Cluster, 6operations, 7Graphite, 7Shinken: Delete specific deployment-prep graphite datapoints - https://phabricator.wikimedia.org/T104091#1575227 (10hashar) [13:46:13] PROBLEM - Host mira is DOWN: CRITICAL - Host Unreachable (10.68.17.215) [14:18:10] 6Release-Engineering: Learn the basics of Puppet - https://phabricator.wikimedia.org/T96560#1575361 (10zeljkofilipin) 5Open>3Resolved Done with the learning VM. [14:27:38] hashar, puppet was disabled on deployment-puppetmaster because it's corrupt [14:28:52] (03CR) 10Hashar: "The first run for this JJB based job can be seen at https://integration.wikimedia.org/ci/job/apps-android-wikipedia-emulator/2/" [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [14:35:33] Yippee, build fixed! [14:35:34] Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #240: FIXED in 7 min 33 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/240/ [14:46:28] 10Scap3: Scap3 service sync should break up remote deploy tasks - https://phabricator.wikimedia.org/T109861#1575503 (10demon) a:5demon>3None [14:46:32] 10Scap3: Improve scap's output - https://phabricator.wikimedia.org/T109858#1575506 (10demon) a:3demon [15:18:47] (03CR) 10Niedzielski: "@Hashar, thanks for putting this together. Would it be possible to merge your changes into this patch[0]? I have another patch[1] that dep" [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [15:21:12] Yippee, build fixed! [15:21:12] Project beta-update-databases-eqiad build #2463: FIXED in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2463/ [15:25:17] Yippee, build fixed! [15:25:18] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #549: FIXED in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/549/ [15:25:34] (03Abandoned) 10Hashar: Create apps-android-wikipedia-emulator [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [15:27:38] 10Continuous-Integration-Infrastructure, 6operations: Update RDiscount gem/package on jenkins build servers (UbuntuTrusty) - https://phabricator.wikimedia.org/T109005#1575709 (10cscott) Go ahead! I was just guesing when I categorized it. [15:29:44] 10Continuous-Integration-Config, 6operations: Switch CI from jsduck deb package to a gemfile/bundler system - https://phabricator.wikimedia.org/T109005#1575715 (10cscott) [15:31:20] (03CR) 10Hashar: "I have deleted https://integration.wikimedia.org/ci/job/apps-android-wikipedia-emulator/" [integration/config] - 10https://gerrit.wikimedia.org/r/233923 (https://phabricator.wikimedia.org/T110307) (owner: 10Hashar) [15:32:05] thcipriani: bah Stephen already crafted a JJB configuration patch on https://gerrit.wikimedia.org/r/#/c/230260/5/jjb/mobile.yaml :-} [15:34:38] (03CR) 10Hashar: [C: 04-1] "I have a few comments coming from my experiment this morning on https://gerrit.wikimedia.org/r/#/c/233923/3/jjb/mobile.yaml,unified all t" (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/230260 (https://phabricator.wikimedia.org/T62720) (owner: 10Niedzielski) [15:35:26] 10Continuous-Integration-Infrastructure, 5MW-1.26-release, 5Patch-For-Review: Fetch dependencies using composer instead of cloning mediawiki/vendor for non-wmf branches - https://phabricator.wikimedia.org/T90303#1575757 (10JanZerebecki) [15:35:36] 10Continuous-Integration-Config, 10Wikibase-JavaScript-Api, 10Wikidata: Make Wikibase-JavaScript-API testextension job run composer - https://phabricator.wikimedia.org/T110172#1570510 (10JanZerebecki) [15:36:43] 10Continuous-Integration-Config, 10Wikibase-JavaScript-Api, 10Wikidata: Make Wikibase-JavaScript-API testextension job run composer - https://phabricator.wikimedia.org/T110172#1570510 (10JanZerebecki) [15:38:17] 10Continuous-Integration-Config, 10Wikibase-JavaScript-Api, 10Wikidata: Make Wikibase-JavaScript-API testextension job run composer - https://phabricator.wikimedia.org/T110172#1575776 (10JanZerebecki) a:5JanZerebecki>3None [15:49:19] hashar: yeah, FWIW that vm.heapSize thing (512 and 256) on the android job was me testing xml output from the jjb patch: can probably ditch that. [15:53:35] thcipriani: probably. I reviewed is change and pointed him to this channel :-D [15:53:42] meanwhile [15:53:48] I gave Grafana a try last night [15:54:00] and figured out a lame dashboard for Gerrit activity https://grafana.wikimedia.org/#/dashboard/db/releng-gerrit ( ostriches ) [15:54:34] will probably craft one for Zuul at https://grafana.wikimedia.org/#/dashboard/db/releng-zuul [15:54:46] feel free to add some more graphs / metrics :} [15:56:37] ERROR: Error fetching remote repo 'origin' [15:56:37] hudson.plugins.git.GitException: Failed to fetch from git://zuul.eqiad.wmnet/mediawiki/core [15:56:51] on integration-slave-precise-1014 [15:57:03] https://integration.wikimedia.org/ci/job/mediawiki-core-phplint/9463/console [16:01:46] !log sudo rm -rf integration-slave-precise-1014:/mnt/jenkins-workspace/workspace/mediawiki-core-phplint/.git [16:01:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:03:15] PROBLEM - Puppet failure on integration-slave-jessie-1001 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:03:16] PROBLEM - Puppet failure on deployment-mx is CRITICAL 100.00% of data above the critical threshold [0.0] [16:03:16] PROBLEM - Puppet failure on deployment-parsoidcache02 is CRITICAL 100.00% of data above the critical threshold [0.0] [16:07:14] oh boy, i love the smell of puppet failure in the morning [16:07:49] PROBLEM - Free space - all mounts on deployment-fluorine is CRITICAL deployment-prep.deployment-fluorine.diskspace.root.byte_percentfree (<100.00%) [16:10:33] !log disk space at 97% on deployment-flourine, mainly due to 15G /var/log/udp2log/udp2log.log [16:10:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:11:47] Something is pretty messed up with git clone of git://zuul.eqiad.wmnet/mediawiki/core to integration-slave-precise-1014 [16:12:24] I'm trying to clone that manually into my homedir just to test and it seems to have hung badly [16:12:35] jenkins isn't having any better luck [16:13:18] the clone got to "Resolving deltas: 100% (478160/478160), done." and has been stuck there for quite a while [16:14:58] !log udp2log is mostly "egrep: writing output: Broken pipe" [16:15:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:15:40] marxarelli: hmmm... did you kill and restart udp2log yet? [16:15:49] haven't, no [16:16:22] !log stopping udp2log on deployment-flourine [16:16:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:16:55] udp2log is a weird beast. c program that spawns python/shell scripts to move data around. I've seen it blow up in lots of interesting ways [16:17:03] bd808: can i safely delete or rotate udp2log.log? [16:17:29] I'd down the service, nuke the log and start the service back up just to be safe [16:18:34] !log deleted udp2log.log and restarted service. so far nothing out of `tail -fn0 udp2log.log` [16:27:05] PROBLEM - Puppet failure on deployment-kafka02 is CRITICAL 22.22% of data above the critical threshold [0.0] [16:28:04] PROBLEM - Puppet failure on integration-slave-precise-1013 is CRITICAL 66.67% of data above the critical threshold [0.0] [16:28:06] PROBLEM - Puppet failure on deployment-db2 is CRITICAL 22.22% of data above the critical threshold [0.0] [16:31:13] thcipriani: So, `scap` is still failing on staging. [16:31:23] At rsync common, it yells at the sudo bit [16:31:35] And then, predictably, permission failures through the rest [16:32:00] yeah, I removed the puppet patch that puts the servicedeploy user on the restbase boxes [16:32:37] that'd probably do it. Also, I think there was at least one place where the config user doesn't get passed to the instance correctly...lemme futz real quick [16:32:52] This is just base scap, not restbase bits. [16:33:08] * ostriches wants that working too, so we can at least make sure our changes aren't causing regressions [16:33:35] oh...wanna jump on deployment-systems triage hangout real quick, twentyafterfour and I are working through some scap stuff right now, actually. [16:33:55] Also, host-key-verification failed. [16:34:03] Yeah, I should be there. [16:34:45] mediawiki-core-phplint FAILURE in 10m 00s [16:35:02] https://integration.wikimedia.org/ci/job/mediawiki-core-phplint/9468/console [16:35:04] just timed out? [16:36:17] legoktm: that's the git clone stuff I was whining about in backscroll :/ [16:36:28] * legoktm looks up [16:36:33] maybe just 1014 is borked? [16:37:17] maybe, it's a brand new slave: https://phabricator.wikimedia.org/T110052#1575146 [16:38:04] can we just mark it offline for now? [16:38:26] https://integration.wikimedia.org/ci/job/tox-flake8/8196/console [16:38:27] yeah [16:38:31] it's borked [16:38:35] i'll do that [16:39:17] !log marked https://integration.wikimedia.org/ci/computer/integration-slave-precise-1014/ offline for git clone problems [16:39:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:40:22] thanks [16:40:45] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1575973 (10bd808) >>! In T110052#1575146, @hashar wrote: > Zeljko r... [16:40:51] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1575975 (10Legoktm) >>! In T110052#1575146, @hashar wrote: > Zeljko... [16:41:01] ha [16:41:26] edit conflict! :) [16:47:11] greg-g: finished the lame Zuul/CI dashboard on Grafana https://grafana.wikimedia.org/#/dashboard/db/releng-zuul [16:47:19] greg-g: guess we will want to discuss it next tuesday [16:49:29] 10Continuous-Integration-Infrastructure, 6Labs, 10Labs-Infrastructure: integration-slave-trusty-1014 and integration-slave-trusty-1017 instances can't boot anymore, ended up corrupted. Need rebuild - https://phabricator.wikimedia.org/T110052#1575990 (10hashar) Seems it takes more than 10 minutes to clone med... [16:53:10] hashar_: sweet, yeah, we'll chat more [16:53:14] I should be ignoring IRC :) [16:53:38] bd808: QUESTION [16:53:55] How do the symlinks in /usr/local/bin for scap-related utils get setup? [16:55:22] Or, perhaps they shouldn't. scap has a hardcoded path tho :p [16:55:41] the symlinks were old cruft [16:56:01] things should use the /srv/deployment/scap/scap/bin paths [16:57:55] That's what I thought [16:58:55] utils.sudo_check_call('l10nupdate', [16:58:55] '/srv/deployment/local/bin/refreshCdbJsonFiles ' [16:58:55] '--directory="%s" --threads=%s %s' % ( [16:58:57] cache_dir, use_cores, verbose_messagelist), [16:58:59] logger) [16:59:05] Fails for obvious reasons in non-crufty environments :) [16:59:36] Bleh, /usr/local/bin [16:59:44] You caught me half-hacked. [16:59:46] yuck. there's a function that should be used to generate the path there based on config [17:01:09] cli.get_script_path [17:01:52] Which...I don't think tasks has access to [17:01:57] yeah :/ [17:02:11] config['bin_dir'] is the magic [17:02:18] which can be handed into a task [17:03:32] cfg is already passed to that function [17:09:01] 6Release-Engineering: Learn the basics of Puppet - https://phabricator.wikimedia.org/T96560#1576021 (10zeljkofilipin) 5Resolved>3Open [17:10:11] 6Release-Engineering: Learn the basics of Puppet - https://phabricator.wikimedia.org/T96560#1220631 (10zeljkofilipin) As a quick exercise, let's try converting the shell script for selenium-vagrant to puppet. https://github.com/zeljkofilipin/selenium-vagrant/commits/puppet [17:50:25] 6Release-Engineering: Learn the basics of Puppet - https://phabricator.wikimedia.org/T96560#1576226 (10greg) >>! In T96560#1576021, @zeljkofilipin wrote: > As a quick exercise, let's try converting the shell script for selenium-vagrant to puppet. > > https://github.com/zeljkofilipin/selenium-vagrant/commits/pup... [17:55:32] ostriches: do you want a patch for that scap path problem? [17:57:09] I hadn't gotten to one yet [17:57:51] (03PS1) 10BryanDavis: Use configured bin_dir to find refreshCdbJsonFiles [tools/scap] - 10https://gerrit.wikimedia.org/r/234028 [17:57:58] ostriches: ^ [17:59:29] Testing [18:01:40] 6Release-Engineering, 6Reading-Infrastructure-Team, 10Sentry, 7Blocked-on-Operations, and 3 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1576289 (10akosiaris) Not sure were we are with this. @Tgr apart from T96054 which I 'll comment on, is there anything else that... [18:02:40] ostriches: Is 1.26wmf22 going to be the last before 1.27wmf0? [18:02:49] Good question. [18:03:04] oh crap :/ [18:03:25] (03CR) 10Chad: [C: 032] Use configured bin_dir to find refreshCdbJsonFiles [tools/scap] - 10https://gerrit.wikimedia.org/r/234028 (owner: 10BryanDavis) [18:03:34] bd808: thx. I'll merge to scap3 branch too [18:03:45] (03Merged) 10jenkins-bot: Use configured bin_dir to find refreshCdbJsonFiles [tools/scap] - 10https://gerrit.wikimedia.org/r/234028 (owner: 10BryanDavis) [18:04:26] thcipriani: scap3 includes it now [18:07:04] ostriches: neat. [18:07:36] Only failure with scap on staging now is the host key thingie, but we knew that [18:08:09] there's a local patch on deployment-bastion to fix that; shouldn't be merged but could be borrowed [18:09:00] Just cleared the known_host file for mwdeplo [18:09:04] hehe, #cheated :p [18:10:10] ostriches: the cherry-pick is https://gerrit.wikimedia.org/r/#/c/148112/ [18:10:41] Could be made a config thing pretty easily [18:11:43] Still failing. Bleh [18:11:51] Not with that patch, but after clearing known_hosts [19:03:33] 6Release-Engineering, 6Reading-Infrastructure-Team, 10Sentry, 7Blocked-on-Operations, and 3 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1576572 (10Tgr) Only code review. The patch does include a puppet role to provision a DB. [19:45:42] 10Browser-Tests, 6Release-Engineering: Create slides about CI browser tests for Metrics meeting - https://phabricator.wikimedia.org/T110385#1576738 (10greg) 3NEW a:3dduvall [19:54:06] 6Release-Engineering, 6Reading-Admin, 6Zero, 7Mobile, 7Technical-Debt: Pull WikipediaMobileFirefoxOS from mediawiki-config - https://phabricator.wikimedia.org/T107172#1576775 (10dr0ptp4kt) By the way, I recommend against removing it from mediawiki-config. Wikipedia for Firefox OS users depend upon the co... [20:00:44] 6Release-Engineering: Create slides about CI browser tests for Metrics meeting - https://phabricator.wikimedia.org/T110385#1576799 (10dduvall) [20:12:13] bd808: l10n cache is confusing me again... https://gerrit.wikimedia.org/r/#/c/232930/ went out with wmf20 (we weren't loading that file previously), but Gadget namespaces are still unlocalized in https://fa.wikivoyage.org/w/api.php?action=query&meta=siteinfo&siprop=namespaces&uselang=en, do you have any ideas why? [20:13:32] legoktm: you are mean for making me think about l10nupdate :/ [20:13:40] :( [20:13:58] I see 'GadgetsNamespaces' in ExtensionMessages-1.26wmf20.php [20:14:48] 10Beta-Cluster, 10Fundraising Tech Backlog, 10MediaWiki-extensions-CentralNotice, 3Fundraising Sprint Rowlf the Dog: Create meta.m.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T110273#1576846 (10Krenair) 5Open>3Resolved I just did https://gerrit.wikimedia.org/r/#/c/234024/, set up *.w... [20:15:38] hmmm... [20:16:24] this is also the first extension that is loading namespaces through extension.json, so it might be going wrong there [20:17:01] that may be the thing to look into first [20:17:27] Like to see if l10nupdate actually removed things when it ran [20:17:38] not sure that we really log that in any way [20:19:08] ok, I can reproduce locally, so it's probably extension.json [20:20:09] have you taught LocalisationUpdate about extension.json? [20:20:31] * bd808 pokes around to remember how this works [20:21:42] It shouldn't have to know...if that was broken we would have noticed a long time ago with special page aliases being broken too [20:23:42] wait... does l10nupdate even look at namesapces? [20:24:14] LU_ReaderFactory doesn't have anything for *.namespace.php [20:26:28] yeah l10nupdate never thinks about $namespaceNames [20:26:41] so the problem is in core somewhere if it exists [20:31:25] bd808, why is mira.deployment-prep.eqiad.wmflabs not prefixed with deployment-? [20:31:38] because reasons [20:32:09] when I made it I thought that the new project component of the hostname was canonical [20:32:18] but later I found out it is not [20:32:20] ah [20:33:24] legoktm: I'm confused about *.namespaces.php and LocalisationCache in general now... [20:34:00] I've been looking through the list of deployment-prep instances, it's quite a mess :/ [20:34:18] legoktm: see https://github.com/wikimedia/mediawiki/blob/master/includes/cache/LocalisationCache.php#L861-L865 [20:35:30] What does deployment-salt do these days? [20:35:36] I notice it's the puppetmaster for some instances [20:35:44] Isn't that supposed to be deployment-puppetmaster? [20:36:32] it is supposed to be deployment-puppetmaster salt is saltmaster in deployment-prep these days. [20:36:38] I think the settings in the ldap are clobbered by settings in hiera. salt is just the saltmaster now [20:37:08] Ugh. [20:37:25] I'm pretty sure this wins -- https://wikitech.wikimedia.org/wiki/Hiera:Deployment-prep [20:38:04] over things in ldap and things in the actual puppet repr [20:41:51] legoktm: I'm afraid if I stay in the maze of twisty passages that is the l10n cache I will be eaten by a grue. :(( [20:43:16] ok :P [20:54:22] 6Release-Engineering: Investigate https://scrutinizer-ci.com/pricing WMF logo use - https://phabricator.wikimedia.org/T110396#1577008 (10greg) 3NEW a:3greg [21:10:37] (03PS1) 10Jdlrobson: Add Jenkins jobs for Quick Surveys [integration/config] - 10https://gerrit.wikimedia.org/r/234141 [21:11:17] (03PS2) 10Jdlrobson: Add Jenkins jobs for Quick Surveys [integration/config] - 10https://gerrit.wikimedia.org/r/234141 [21:11:23] (03PS3) 10Jdlrobson: Add Jenkins jobs for Quick Surveys [integration/config] - 10https://gerrit.wikimedia.org/r/234141 (https://phabricator.wikimedia.org/T110402) [21:16:28] bd808: so far I've tracked it down to https://www.mediawiki.org/wiki/Special:Code/MediaWiki/85327 breaking the documented way to do it. [21:16:33] it = extensions adding namespaces [21:17:42] legoktm: o_0 so for 4.5 years nobody noticed? [21:18:06] bug 1 strikes again [21:38:10] Project beta-update-databases-eqiad build #2470: FAILURE in 16 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2470/ [21:43:32] Yippee, build fixed! [21:43:33] Project beta-update-databases-eqiad build #2471: FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2471/ [21:45:05] I'm fixing a Flow issue with the Beta databases (due to a typo in this script, since fixed). I can't see why https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2470/consoleFull failed, though. [21:50:13] Project beta-update-databases-eqiad build #2472: FAILURE in 10 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2472/ [21:50:59] !log To fix https://gerrit.wikimedia.org/r/#/c/233952/1 on Beta, manually ran: while read line; do echo "Starting $line\n"; echo 'ALTER TABLE flow_wiki_ref DROP COLUMN ref_src_wiki;' | sql --write "$line"; echo "Finished $line\n"; done < /srv/mediawiki/all-labs.dblist [21:51:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [22:02:49] 10Beta-Cluster: Output of wmf-beta-update-databases.py is not clear on errors - https://phabricator.wikimedia.org/T110407#1577237 (10Mattflaschen) 3NEW [22:08:04] Well, I figured out the Flow bug. However, that bug (doesn't pass along errors) stands. [22:21:09] Yippee, build fixed! [22:21:09] Project beta-update-databases-eqiad build #2474: FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/2474/ [22:41:22] 6Release-Engineering: Investigate https://scrutinizer-ci.com/pricing WMF logo use - https://phabricator.wikimedia.org/T110396#1577411 (10hashar) We have three repositories from the Github wikimedia/ umbrella: https://scrutinizer-ci.com/g/wikimedia/mediawiki/ https://scrutinizer-ci.com/g/wikimedia/mediawiki-skin... [22:47:44] Yippee, build fixed! [22:47:45] Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #779: FIXED in 36 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/779/ [23:09:45] (03CR) 10Dduvall: [C: 04-1] "Small nitpick over indentation but looks good otherwise." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/234141 (https://phabricator.wikimedia.org/T110402) (owner: 10Jdlrobson) [23:26:21] Can jenkins-bot add a Verified-2 or is there only Verified-1? (I think the latter, but not 100% sure) [23:26:45] (because when the Basic test fails the main test is probably not tried anymore?) [23:29:01] think it depends on repo config? [23:30:27] I also think so. :-/ [23:31:04] just wondering whether to cover V-2 also in Tech-Community-Metrics on korma [23:31:22] Currently there are no V-2 changesets in Gerrit but wondering if there ever could be and whether to cover that [23:32:31] !log Re-armed keyholder on deployment-bastion [23:32:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [23:39:26] !log Updated scap to a7ec319 (Use configured bin_dir to find refreshCdbJsonFiles) [23:39:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master