[02:17:13] Yippee, build fixed! [02:17:13] Project selenium-QuickSurveys » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #111: 09FIXED in 4 min 12 sec: https://integration.wikimedia.org/ci/job/selenium-QuickSurveys/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/111/ [06:40:20] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:58:17] PROBLEM - Puppet run on phab-beta is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [07:05:19] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:05:09] PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 567 bytes in 0.002 second response time [08:06:17] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 301 TLS Redirect - string 'Wikipedia' not found on 'http://en.m.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 569 bytes in 0.003 second response time [08:31:48] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2532058 (10phuedx) @Jdlrobson, @dr0ptp4kt: Should we be diving into... [09:02:33] 07Browser-Tests, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, 07Regression, 07Unplanned-Sprint-Work: [Regression] Fix browser tests for language switching on the beta cluster - https://phabricator.wikimedia.org/T141647#2532133 (10phuedx) [The last four builds are green](https://integ... [09:32:36] 10Beta-Cluster-Infrastructure, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] (tracking) - https://phabricator.wikimedia.org/T142288#2532178 (10Ladsgroup) > -ores-redis (@Ladsgroup) into -redis0[12]? We decided to have as similar as possible to prod... [10:13:36] 03Scap3: TypeError: unsupported operand type(s) for %: 'dict' and 'tuple' - https://phabricator.wikimedia.org/T142364#2532315 (10Ladsgroup) [10:14:46] !log deploying 616707c into sca03 (for ores) [10:14:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [10:31:42] 10Beta-Cluster-Infrastructure, 10Citoid, 10VisualEditor: Move citoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142150#2532356 (10mobrovac) 05Open>03Invalid Citoid is already being deployed to `deployment-sca0[12]`. Closing as invalid. [11:29:12] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [11:29:15] 10Beta-Cluster-Infrastructure, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] (tracking) - https://phabricator.wikimedia.org/T142288#2532425 (10mobrovac) >>! In T142288#2531552, @AlexMonk-WMF wrote: > Delete one or two -parsoid (@Catrope, @ssastry,... [11:29:58] PROBLEM - Host deployment-parsoid07 is DOWN: CRITICAL - Host Unreachable (10.68.21.30) [11:30:32] PROBLEM - Host deployment-parsoid06 is DOWN: CRITICAL - Host Unreachable (10.68.19.232) [11:33:29] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 06Services, 07Epic: Meeting: Automatic deployment of backend services on beta cluster - https://phabricator.wikimedia.org/T100099#2532439 (10Danny_B) [11:42:26] 10MediaWiki-Releasing, 06Release-Engineering-Team, 06MediaWiki-Stakeholders-Group: Establish and consistently release schedule for MediaWiki releases - https://phabricator.wikimedia.org/T90249#2532472 (10Danny_B) [11:50:29] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:01:44] Yippee, build fixed! [12:01:44] Project selenium-RelatedArticles » chrome,beta-desktop,Linux,contintLabsSlave && UbuntuTrusty build #106: 09FIXED in 43 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-desktop,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/106/ [12:01:50] Yippee, build fixed! [12:01:51] Project selenium-RelatedArticles » chrome,beta-mobile,Linux,contintLabsSlave && UbuntuTrusty build #106: 09FIXED in 50 sec: https://integration.wikimedia.org/ci/job/selenium-RelatedArticles/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta-mobile,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/106/ [12:23:02] Yippee, build fixed! [12:23:03] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #107: 09FIXED in 1 min 2 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/107/ [13:44:14] hey, does the beta cluster have varnish (or similar like squid) cache? [13:45:02] 05Gerrit-Migration, 10Differential, 07Documentation: Document use of Owners in Phabricator and advertise it - https://phabricator.wikimedia.org/T128372#2533060 (10Aklapper) FYI I've created a stub section about "Owners" on https://www.mediawiki.org/wiki/Phabricator/Differential [14:39:50] !log deploying d00159c for ores in sca03 [14:39:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [14:43:51] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2533188 (10dr0ptp4kt) @phuedx yes, go for it. [15:20:19] 03Scap3 (Scap3-Adoption-Phase1), 03releng-201516-q4: [keyresult] Migrate remaining trebuchet deployed services - https://phabricator.wikimedia.org/T129290#2533452 (10Nuria) [15:20:22] 10Deployment-Systems, 03Scap3 (Scap3-Adoption-Phase1), 10scap, 10Analytics-Cluster, and 2 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2533451 (10Nuria) 05Open>03Resolved [15:21:45] Amir1, yes [15:22:04] thanks [15:28:26] Amir1, mobrovac: thanks for your input re. deployment-prep instance sizes [15:30:33] yw :) [15:31:03] np Krenair [15:40:49] Krenair it seems shinken-wm (~shinken-w@internal-server-nat.wmflabs.org) [15:40:51] is being killed [15:41:02] by Sigyn [15:41:06] * shinken-wm has quit (Killed (Sigyn (Spam is off topic on freenode.))) [15:41:23] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team, 10DBA, 07WorkType-Maintenance: Upgrade mariadb in deployment-prep from Precise/MariaDB 5.5 to Jessie/MariaDB 5.10 - https://phabricator.wikimedia.org/T138778#2409864 (10AlexMonk-WMF) I've seen a new instance called deployment-db01... But it should... [15:41:28] I know paladox [15:41:32] oh [15:41:33] see -labs [15:41:36] ok [15:41:38] thanks [15:52:48] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2533561 (10greg) >>! In T1... [16:11:04] mobrovac, maybe https://phabricator.wikimedia.org/T142150 meant -zotero01 instead? [16:11:50] Krenair: likely not, greg-g and i already discussed -zotero01 [16:12:00] on which task? [16:12:10] but he can reopen it if it indeed was a lapsus linguae [16:19:45] !log Manually cleaned up root@logstash02 cronjobs related to logstash03 [16:19:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [16:20:24] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2533710 (10greg) p:05Unbreak!>03High It's not happening across instances consistently. It's not happening... [16:21:48] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:23:46] 10Continuous-Integration-Infrastructure, 10MediaWiki-Unit-tests, 07Regression: Job mediawiki-extensions-php55 frequently fails due to "Segmentation fault" - https://phabricator.wikimedia.org/T142158#2525383 (10bd808) >>! In T142158#2525400, @Legoktm wrote: > Do we have a core dump? Getting the core file is... [16:52:20] Krenair go with what mobrovac says, I was probably misremembering as I did that task creation at something 1am my time :) [16:52:28] somethig like* [16:53:33] ok [17:00:22] (03PS1) 10Nemo bis: Finish removing MoodBar, including nl.wikipedia [tools/release] - 10https://gerrit.wikimedia.org/r/303575 (https://phabricator.wikimedia.org/T131340) [17:00:47] 05Gerrit-Migration, 10Differential, 07Documentation: Document use of Owners in Phabricator and advertise it - https://phabricator.wikimedia.org/T128372#2533947 (10mmodell) @aklapper: I've expanded it just a bit to mention audit and provide a link to the "create package" form. I think we are in pretty good sh... [17:02:21] (03CR) 10MarcoAurelio: [C: 031] Finish removing MoodBar, including nl.wikipedia [tools/release] - 10https://gerrit.wikimedia.org/r/303575 (https://phabricator.wikimedia.org/T131340) (owner: 10Nemo bis) [17:02:39] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2533951 (10greg) This table should migrate to a wiki page on mw.org. @zeljkofilipin ? :) [17:07:36] 07Browser-Tests, 06Release-Engineering-Team, 07Documentation: Document browser tests ownership (and what it means) on wiki - https://phabricator.wikimedia.org/T142409#2533979 (10greg) [17:08:08] 07Browser-Tests, 06Release-Engineering-Team, 07Documentation: Document browser tests ownership (and what it means) on wiki - https://phabricator.wikimedia.org/T142409#2533993 (10greg) [17:08:26] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2267231 (10greg) [17:08:40] 10Browser-Tests-Infrastructure, 13Patch-For-Review, 15User-zeljkofilipin: Ownership of Selenium tests - https://phabricator.wikimedia.org/T134492#2267231 (10greg) >>! In T134492#2533951, @greg wrote: > This table should migrate to a wiki page on mw.org. @zeljkofilipin ? :) Made {T142409} [17:25:26] greg-g I finished the rest of the gerrit puppet update to support it on labs at the weekend [17:25:36] was pretty easy since we can rely on labs https [17:25:55] awesome [17:26:00] instead of trying to create a https certificate [17:26:03] i have a running [17:26:12] https://gerrit-test3.wmflabs.org/ [17:26:30] ldaps works too [17:27:20] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 2 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2534092 [17:28:28] It allows all users who use wikitech to login [17:28:29] :) [17:28:58] Puppet is disabled due to me trying the change before i uploaded to gerrit and awiting review :) [17:31:22] 10Beta-Cluster-Infrastructure, 06Commons, 10MediaWiki-Authentication-and-authorization, 10MediaWiki-extensions-CentralAuth, 06Reading-Web-Backlog: Unable to log in on https://commons.m.wikimedia.beta.wmflabs.org/wiki/Special:UserLogin - https://phabricator.wikimedia.org/T142015#2534112 (10matmarex) I don... [17:38:01] twentyafterfour i got the its-phabricator working [17:38:13] You have to get the token from a different place [17:39:51] twentyafterfour you get the token from [17:39:51] https://phab-01.wmflabs.org/conduit/token/ [17:40:11] then you use https://phab-01.wmflabs.org/conduit/method/conduit.getcertificate/ to get the certificate [17:41:38] That sounds like stuff to document in a single comment in a related Phab task instead? :) [17:41:48] There is no related task [17:41:56] ah. heh. [17:41:58] andre__ but i did make qchris aware of that [17:42:25] Since docs were outdated and took me forever to find out where we get the certificates from [17:43:27] andre__ one thing though you carnt do it that way for a bot, you have to use a user account [17:43:49] * andre__ has to leave now, sorry [17:43:54] Ok [17:43:59] but happy to hear you found out! :) [17:44:19] Yep and thanks. [17:52:00] paladox: there's no task regarding making gerrit run on a labs host? there should be to document this work. [17:59:29] 07Browser-Tests, 06Release-Engineering-Team, 07Documentation: Document browser tests ownership (and what it means) on wiki - https://phabricator.wikimedia.org/T142409#2534225 (10adrianheine) [18:06:13] greg-g oh there is for gerrit [18:06:22] but there isent for its-phabricator [18:06:46] greg-g https://phabricator.wikimedia.org/T141803 [18:22:02] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2534358 (10Tgr) >>! In T142141#2529539, @Jdlrobson wrote: > All are... [18:32:21] 07Browser-Tests, 10MediaWiki-extensions-MultimediaViewer: MultimediaViewer tests fail with waiting for r {:class=>"mw-mmv-final-image"} - https://phabricator.wikimedia.org/T142423#2534386 (10Jdlrobson) [18:32:23] 10Browser-Tests-Infrastructure, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: Upgrade Selenium gem for various reading web opened extensions - https://phabricator.wikimedia.org/T142141#2534399 (10Jdlrobson) Looking closely at the Sauce labs url it's not... [18:32:41] 07Browser-Tests, 10MediaWiki-extensions-MultimediaViewer: MultimediaViewer tests fail with waiting for {:class=>"mw-mmv-final-image"} (Firefox only) - https://phabricator.wikimedia.org/T142423#2534386 (10Jdlrobson) [18:32:59] 07Browser-Tests, 10MediaWiki-extensions-MultimediaViewer: MultimediaViewer tests fail with waiting for {:class=>"mw-mmv-final-image"} (Firefox only) - https://phabricator.wikimedia.org/T142423#2534386 (10Jdlrobson) [18:35:07] greg-g: concerning UBN! issues should I wait for swat or can I deploy it outside swat windows? [18:56:02] Yippee, build fixed! [18:56:03] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #102: 09FIXED in 23 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/102/ [18:57:42] Amir1: just saw https://phabricator.wikimedia.org/T142364 looked at deploy-log, looks like you got deploy to work. What did you have to change? [18:58:17] thcipriani: it was just it and it worked without an issue afterwards [18:58:30] I mean it didn't stopped the script [18:58:41] but kept happening in different deploys [18:58:42] oh, just dumped an ugly error and worked anyway? [18:58:49] yup [18:59:12] hmm I've seen this pop up since the new version, but haven't been able to recreate in development yet. [18:59:42] it happened in beta to me [18:59:50] if that helps [18:59:50] Yippee, build fixed! [18:59:51] Project selenium-CirrusSearch » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #111: 09FIXED in 54 sec: https://integration.wikimedia.org/ci/job/selenium-CirrusSearch/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/111/ [19:00:41] yarp. I'll dig into it. [19:33:51] 03Scap3: TypeError: unsupported operand type(s) for %: 'dict' and 'tuple' - https://phabricator.wikimedia.org/T142364#2534677 (10thcipriani) [20:12:30] greg-g: thcipriani I think deployment issues in -ops? [20:12:35] re: subbu [20:12:41] * thcipriani looks [20:13:30] trebuchet/salt.... [20:18:11] greg-g i plan on also fixing phabricator puppet roles in labs. [20:18:33] Making scap optional too, to prevent failures due to labs not supporting what scap needs to work. [20:19:28] greg-g: so we do thse ops sessions which is like a state of the union address on how some thing works [20:19:49] maybe you could guys could do one on deploy options / scap3 [20:19:59] I realize I have no clue where things stand and how they work or better yet even should work :0 [20:21:51] chasemp: heh, you're probably not the only one. Mostly things that are deployed should use scap3, parsoid is moving Soon™. I know how Trebuchet works fairly well, but I'd rather not have to :P [20:22:18] there was a lot of parsoid puppet work to move it over to use service::node so the scap3 transition would be painless. That work just finished up. [20:22:55] I would be interested in a guided tour so I can at least ping the right ppl and do a bit of digging when needed [20:23:22] which of the systems is used when the docs include "git deploy" [20:23:29] yeah, I'd be happy to give a guided tour. [20:23:32] mutante: that's trebuchet [20:23:50] ok [20:25:03] we've made a lot of progress over the past quarter moving things to scap3. AFAICT it's mostly parsoid, restbase, a few cassandra-related things, and things that no one really deploys very much that still need to be moved. [20:25:29] https://gerrit.wikimedia.org/r/#/c/303435/ :) [20:25:52] yeah, other than those that tyler just listed, if it's been deployed in the last month, it's on scap3 :) [20:26:52] so it seems that trebuchet is trying to get deployment_config from the pillar [20:27:11] pillar as in "Pillar is an interface for Salt designed to offer global values that can be distributed to all minions." [20:27:26] but that fails [20:28:25] there might be a pillar sync that needs to happen between master and minion? [20:29:28] (although I thought pillar stuff was master-specific) [20:33:51] 06Release-Engineering-Team, 06Operations, 15User-greg, 07Wikimedia-Incident: Institute a weekly review of all UBN! tasks - https://phabricator.wikimedia.org/T141130#2534854 (10greg) 05Open>03Resolved >>! In T141130#2531402, @Aklapper wrote: >>>! In T141130#2515799, @greg wrote: >> Should we re-instate... [20:37:19] 06Release-Engineering-Team, 03releng-201617-q1, 15User-greg, 15User-zeljkofilipin: Perform a technical debt analysis of software and services maintained by WMF Release Engineering - https://phabricator.wikimedia.org/T138225#2534866 (10greg) Based on the updated process proposed by @dduvall we've identified... [20:55:16] uhhh [20:55:38] nvm me [21:05:12] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 2 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2534980 [21:36:02] 10Beta-Cluster-Infrastructure, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] (tracking) - https://phabricator.wikimedia.org/T142288#2530145 (10chasemp) this is an awesome task thanks @AlexMonk-WMF and @greg and other folks too//italic text// [21:36:25] //italic text// [21:59:45] 10Beta-Cluster-Infrastructure, 10ORES, 06Revision-Scoring-As-A-Service: Dashboard or pane for ORES service in beta - https://phabricator.wikimedia.org/T142294#2535110 (10Ladsgroup) 05Open>03Resolved [21:59:49] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-ORES, 06Revision-Scoring-As-A-Service: Dashboard or pane for ORES failed jobs on beta - https://phabricator.wikimedia.org/T142119#2535111 (10Ladsgroup) 05Open>03Resolved [22:02:14] 10Beta-Cluster-Infrastructure, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] (tracking) - https://phabricator.wikimedia.org/T142288#2535127 (10dpatrick) >>! In T142288#2531574, @greg wrote: > mediawiki-03 is a security only host. Are you all still... [22:11:56] twentyafterfour, could you take a look at https://phabricator.wikimedia.org/T142246 when you have a chance? [22:12:10] matt_flaschen: looking [22:12:48] Thanks [22:13:29] matt_flaschen: I haven't tried to convert a subproject to a milestone before but it might work [22:13:42] 06Release-Engineering-Team, 06Operations, 15User-greg, 07Wikimedia-Incident: Institute quarterly(?) review of incident reports and follow-up - https://phabricator.wikimedia.org/T141287#2535154 (10greg) a:05greg>03None [22:13:57] 06Release-Engineering-Team, 15User-greg: Pre 2016 offsite "must do"s - https://phabricator.wikimedia.org/T140923#2535155 (10greg) a:05greg>03None [22:16:01] twentyafterfour, if you're worried it might cause corruption, we can skip it and wait until next quarter. But if you think it will probably either work or just give an error, it would be nice. [22:16:49] I think it should be safe (and I can clean up the damage if there is any) but it might just refuse to convert .. I'm about to try it [22:22:01] Yippee, build fixed! [22:22:01] Project selenium-CentralAuth » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #109: 09FIXED in 1 min 59 sec: https://integration.wikimedia.org/ci/job/selenium-CentralAuth/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/109/ [22:22:05] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 2 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2535194 [22:26:35] 10Beta-Cluster-Infrastructure, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] (tracking) - https://phabricator.wikimedia.org/T142288#2535199 (10AlexMonk-WMF) Note: I'm crossing things off my list above as each item is either dealt with or it is made... [22:27:31] greg-g, mind if I totally hijack your ticket by moving my list to the description so everyone can edit it? [22:28:45] 10Beta-Cluster-Infrastructure, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] (tracking) - https://phabricator.wikimedia.org/T142288#2535206 (10greg) >>! In T142288#2535199, @AlexMonk-WMF wrote: > Note: I'm crossing things off my list above as each... [22:29:03] 10Beta-Cluster-Infrastructure, 07Tracking: Consolidate, remove, and/or downsize Beta Cluster instances to help with [[wikitech:Purge_2016]] (tracking) - https://phabricator.wikimedia.org/T142288#2535207 (10greg) [22:29:09] Krenair: heh, too late :) [22:29:09] oh you're going to do it, okay :) [22:32:21] hopefully getting instance creation back tomorrow [22:34:01] I somehow missed https://phabricator.wikimedia.org/T140904 before [22:34:18] (semi-relatedly) [22:36:20] yep... that's what we have to do now to get more resources unless we reduce our usage in T142288 [22:37:14] * greg-g nods [22:48:23] 06Release-Engineering-Team, 06Operations, 10Traffic, 07HTTPS: Retire gerrit.wikimedia.org SSL cert - https://phabricator.wikimedia.org/T142131#2523742 (10Dzahn) deleted cert from public repo. deleted key from private repo. this should be all then [22:52:15] 06Release-Engineering-Team, 06Operations, 10Traffic, 07HTTPS: Retire gerrit.wikimedia.org SSL cert - https://phabricator.wikimedia.org/T142131#2535257 (10Dzahn) 05Open>03Resolved a:03Dzahn /etc/ssl/localcerts and /etc/ssl/private on lead were already cleaned [23:33:07] !log deleted instance deployment-depurate01 [23:33:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master