[00:02:20] 06Release-Engineering-Team, 06Developer-Relations, 06Project-Admins: Clean #Wiki-Release-Team project - https://phabricator.wikimedia.org/T88263#2552327 (10Mglaser) I just came across this again. It looks like the remaining tasks are all assigned to #MediaWiki-Stakeholders-Group, which is good. Thanks for he... [00:16:53] Project selenium-Flow » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #111: 04FAILURE in 52 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/111/ [00:17:16] Yippee, build fixed! [00:17:16] Project selenium-Flow » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #111: 09FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/selenium-Flow/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/111/ [00:23:17] 10Beta-Cluster-Infrastructure, 06Labs, 10Labs-Infrastructure, 07Tracking: Log files on labs instance fill up disk (/var is only 2GB) (tracking) - https://phabricator.wikimedia.org/T71601#727373 (10AlexMonk-WMF) I haven't seen this issue occur in a long while [00:23:49] 10Beta-Cluster-Infrastructure, 06Operations, 07HHVM: Beta-cluster web server fills up /var/log with Apache logs - https://phabricator.wikimedia.org/T75262#755023 (10AlexMonk-WMF) I haven't seen this issue in a long while... [02:03:06] zuul seems to be stuck. Looks like it stopped doing things about 2.5 hours ago [02:08:10] !log Aug 15 02:07:48 labnodepool1001 nodepoold[24796]: Forbidden: Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) [02:08:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:11:16] !log nodepool delete 278848 [02:11:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:11:30] !log nodepool delete 299641 [02:11:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:19:51] !log nodepool delete 301144 [02:19:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:20:19] !log nodepool delete 301282 [02:20:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:20:41] !log nodepool delete 301291 [02:20:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:21:10] !log nodepool delete 301068 [02:21:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:23:25] !log Lots and lots of "AttributeError: 'NoneType' object has no attribute 'name'" errors in /var/log/zuul/zuul.log [02:23:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [02:26:59] Having gotten nodepool unstuck with manual deletes, I think that zuul needs to be restarted now. There are several items in the queue with "Unknown Project" and a constant stream of that error message about NoneType [02:30:33] !log Forced a zuul restart -- https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Restart [02:30:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [04:07:04] Yippee, build fixed! [04:07:05] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #109: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/109/ [12:22:43] Project selenium-GettingStarted » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #114: 04FAILURE in 42 sec: https://integration.wikimedia.org/ci/job/selenium-GettingStarted/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/114/ [13:40:25] 03Scap3: Sequential execution should be per-deployment, not per-phase - https://phabricator.wikimedia.org/T142990#2553251 (10mobrovac) [13:40:42] 03Scap3, 15User-mobrovac: Sequential execution should be per-deployment, not per-phase - https://phabricator.wikimedia.org/T142990#2553263 (10mobrovac) [13:47:30] Project selenium-VisualEditor » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #112: 04FAILURE in 3 min 29 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/112/ [14:33:31] Project selenium-WikiLove » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #113: 04FAILURE in 1 min 30 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/113/ [14:50:18] (03PS2) 10Florianschmidtwelzow: Disallow parenthesis around keywords like clone or require [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/301828 (https://phabricator.wikimedia.org/T116779) [14:52:41] (03PS1) 10Florianschmidtwelzow: Add .idea directory to .gitignore [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/304834 [14:55:27] RECOVERY - Host deployment-parsoid05 is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [14:58:29] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:59:15] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:01:44] PROBLEM - Host deployment-parsoid05 is DOWN: CRITICAL - Host Unreachable (10.68.16.120) [15:04:30] Project beta-code-update-eqiad build #117137: 04FAILURE in 1 min 29 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/117137/ [15:06:08] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:14:38] Yippee, build fixed! [15:14:38] Project beta-code-update-eqiad build #117138: 09FIXED in 1 min 37 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/117138/ [15:23:19] 10Continuous-Integration-Infrastructure (phase-out-gallium), 06Operations, 10hardware-requests: Allocate contint1001 to releng and allocate to a vlan - https://phabricator.wikimedia.org/T140257#2553490 (10thcipriani) >>! In T140257#2491705, @faidon wrote: > I've deliberated this a little bit and honestly my... [15:46:09] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [15:57:56] PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:07:11] Hello again everyone...I'm about to send in my work laptop for repairs, it's the only computer that's still logged into Phabricator, and my phone authentication token is lost since I got a new phone several months ago. Is there any way for me to verify my identity without going to find one of the other WMFers in Minnesota, or waiting until I visit SF? [16:07:53] Alternatively I could just bother James_F every time I need to do something on Phabricator, but I think he wouldn't appreciate that [16:12:43] I guess I have some time, it says my file transfer won't be done until tonight. But I'd like to have a solution lined up. [16:16:21] marktraceur: Mukunda can reset your token, I think. [16:16:38] James_F: Yeah, but I need to verify it's me somehow, is my understanding [16:16:52] Not someone who has stolen my IRC account and Phabricator password [16:16:53] marktraceur: We have video chat. [16:17:00] marktraceur: And telephones. [16:17:06] I have no idea why that wasn't floated last time I was in here [16:17:17] Oh, wait, they told me to go to -devtools. [16:17:21] #anothergoddamnchannel [16:17:29] lol [16:17:41] Because normally the people asking are volunteers who don't necessarily want to share their appearance (and aren't necessarily recognisable to Mukunda or whomever). [16:39:07] (03CR) 10Addshore: [C: 032] "sure" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/304834 (owner: 10Florianschmidtwelzow) [16:43:03] (03Merged) 10jenkins-bot: Add .idea directory to .gitignore [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/304834 (owner: 10Florianschmidtwelzow) [16:49:14] 07Browser-Tests, 06Reading-Web-Backlog, 13Patch-For-Review, 03Reading-Web-Sprint-78-Terminal-Velocity: Various browser tests failing due to login error - https://phabricator.wikimedia.org/T142600#2553862 (10MBinder_WMF) [16:56:29] (03PS1) 1020after4: outline for `scap swat` command line tool [tools/release] - 10https://gerrit.wikimedia.org/r/304855 [16:56:59] (03CR) 10jenkins-bot: [V: 04-1] outline for `scap swat` command line tool [tools/release] - 10https://gerrit.wikimedia.org/r/304855 (owner: 1020after4) [17:00:41] (03PS2) 1020after4: outline for `scap swat` command line tool [tools/release] - 10https://gerrit.wikimedia.org/r/304855 [17:25:38] test failures that look like a CI setup issue -- https://integration.wikimedia.org/ci/job/rake-jessie/56016/console -- https://integration.wikimedia.org/ci/job/npm-node-4/199/console [17:25:42] rsync: change_dir "/castor-mw-ext-and-skins/REL1_23/npm-node-4" (in caches) failed: No such file or directory (2) [17:25:49] 10Deployment-Systems, 03Scap3 (Scap3-MediaWiki-MVP), 10scap, 10MediaWiki-API, and 3 others: Create a script to run test requests for the MediaWiki service - https://phabricator.wikimedia.org/T136839#2554005 (10Anomie) In that case, you don't really need api.php to return it at all. Just put your file in [[... [17:26:19] Anyone besides hashar know about that castor stuff? [17:30:18] ugh, not really. I have an idea of what it is for, but not the specifics of how it works. [17:32:36] 03Scap3, 15User-mobrovac: Sequential execution should be per-deployment, not per-phase - https://phabricator.wikimedia.org/T142990#2554038 (10dduvall) (Following up on @thcipriani's suggestion in our meeting to rename this to `group_size`, but with some explanation of how we might implement this internally.)... [17:35:02] fwiw, castor.integration.wmflabs:/mnt/jenkins-workspace/caches/castor-mw-ext-and-skins/REL1_23/rake-jessie is what it's looking for in that job. [17:45:12] PROBLEM - Puppet run on deployment-db01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [17:49:48] 03Scap3, 15User-mobrovac: Sequential execution should be per-deployment, not per-phase - https://phabricator.wikimedia.org/T142990#2554076 (10dduvall) p:05Triage>03High a:03dduvall [17:51:33] 10Continuous-Integration-Infrastructure, 06Labs: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2554094 (10thcipriani) [17:56:23] 10Continuous-Integration-Infrastructure, 06Labs: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554147 (10thcipriani) [17:57:14] thcipriani: yeah one fails for rake-jessie and one for npm-node-4. Pretty sure it has something to do with backporting to REL1_23. [18:00:10] yeah, I would guess these dirs need to be initialized manually [18:03:48] 10Continuous-Integration-Infrastructure, 06Labs: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554186 (10Paladox) Are we looking for https://github.com/openstack-infra/nodepool/blob/877265912853e0f7a6a21e3771fe9250f71e9c26/nodepool/nodepool.py#L57 has it been configured to be lo... [18:05:02] 10Continuous-Integration-Infrastructure, 06Labs: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2554189 (10thcipriani) [18:06:53] 10Continuous-Integration-Infrastructure, 06Labs: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2554218 (10Paladox) [18:06:55] 10Continuous-Integration-Infrastructure, 06Labs: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554219 (10Paladox) [18:06:58] 10Continuous-Integration-Infrastructure, 06Labs: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2554220 (10Paladox) [18:13:15] thcipriani or twentyafterfour hi could you create a master branch in gerrit just so https://phabricator.wikimedia.org/diffusion/ODEV/ works [18:13:27] or change default branch https://phabricator.wikimedia.org/diffusion/ODEV/ to something that is a branch please? [18:14:22] I think what needs to happen with rake-jessie is we need to run bundle install --clean and set the BUNDLE_PATH to the directory on castor so that the deps are installed there for REL1_23 [18:14:49] Oh wait didnt we move those jobs a few days ago [18:15:00] castor may not work outside of nodepool [18:15:46] eh, I think it should work, castor is in the integration project afaict. None of those jobs are failing except for the branches that don't exist on castor [18:16:05] Oh sorry [18:16:07] wrong test [18:16:36] actually, maybe I can just create those directories and let the jobs themselves populate them. Might take longer initially, but after the initial ending rsync it should go faster. [18:20:45] 10Continuous-Integration-Infrastructure, 06Labs: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554306 (10Paladox) We are using nodepool version 0.1.1, we should update to 0.3.0. [18:36:12] 10Continuous-Integration-Config, 10Fundraising-Backlog: mediawiki/extensions/DonationInterface/vendor repo needs CI V+2 jobs - https://phabricator.wikimedia.org/T143025#2554390 (10awight) [18:41:26] aww nuts, it's possible that I killed zuul by V+2'ing and submitting a patch. See the ^ above bug--is there a way I can submit patches in this repo without leaving the swath of destruction? [18:44:24] (03PS1) 10Awight: Tests for DonationInterface/vendor submodule [integration/config] - 10https://gerrit.wikimedia.org/r/304875 (https://phabricator.wikimedia.org/T143025) [18:45:20] That should fix it... for next time. [18:54:27] 10Continuous-Integration-Infrastructure, 07Nodepool: 2016-08-10 CI incident follow-ups - https://phabricator.wikimedia.org/T142952#2554491 (10greg) [18:58:02] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2554504 (10greg) [18:58:25] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2554506 (10greg) [18:58:33] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2554512 (10greg) [18:59:15] ok, incident follow-up gardening done [19:16:03] hey marxarelli [19:16:33] Krenair: hey [19:17:44] marxarelli, I was wondering if you had done any work on deployment-db01 [19:18:18] i haven't really. i was planning on picking that up today [19:19:44] marxarelli, so I was hoping we could rename it before you invest any time/effort there [19:21:08] Krenair: sure though i don't quite understand what is wrong with the name though. if we do db03 and db04, that also seems confusing after db1 and db2 are terminated [19:21:22] We already have deployment-db1 [19:21:37] We don't reuse host numbers [19:22:01] ah, alright. didn't realize there was a protocol there [19:22:08] no problem then [19:22:34] e.g. parsoid is up to 09 at this point [19:22:43] memc at 05 [19:22:46] caches at 04 [19:22:52] i'll terminate db1 and db2 and start up a db03 and db04 [19:23:02] er, terminate db01 i mean [19:23:30] yeah you didn't make a db02, but there is now space for you to do so :) [19:23:34] thanks [19:23:41] np [19:24:23] oh, my bad, we do have memc04 as well as memc05 [19:24:26] still [19:26:04] db-masterNN and db-replicaNN might be a better naming convention, just so there's no confusing on which is which [19:26:10] * marxarelli is bikeshedding [19:26:21] possibly [19:26:27] I'd run that by Jaime though [19:26:42] It can get more complex than that: https://tendril.wikimedia.org/tree [19:27:43] I think most of the replicas-of-replicas stuff in prod is for codfw/analytics/labs which we don't handle in beta [19:28:33] Krenair: holy cow. preeeettty :) [19:28:37] :D [19:29:14] if we did switch to that we could set up either db-master01+db-replica01 or db-master02+db-replica02 I suppose [19:30:19] with the potential for such a complex setup, perhaps it is better that the name not inspire assumptions :) [19:31:00] so i'll go with your original recommendation [19:31:00] yeah, my only real concern with db-master/replica is that we could do something more complicated in future and the names no longer match [19:31:08] right [19:31:42] i was surprised when i saw that the slave/replica wasn't configured to be globally read-only but it makes sense now [19:35:17] well whatever happens with naming, let me know if I can help with anything. [19:35:53] 10Beta-Cluster-Infrastructure, 06Commons, 06Multimedia: Setup deployment-imagescaler host(s) in Beta Cluster - https://phabricator.wikimedia.org/T142289#2554799 (10greg) >>! In T142289#2539296, @Gilles wrote: > The first step will be to have thumbor receive the same requests as the existing image scaler(s) a... [19:41:39] thanks, Krenair. will do [19:44:33] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-Terminal-Velocity, and 2 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, Relat... - https://phabricator.wikimedia.org/T129483#2554816 [19:45:05] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2554821 [19:47:44] 06Release-Engineering-Team, 06Labs, 06Operations, 10wikitech.wikimedia.org, 07LDAP: Rename specific account in LDAP, Wikitech and Gerrit - https://phabricator.wikimedia.org/T133968#2554836 (10demon) [20:42:07] Yippee, build fixed! [20:42:07] Project selenium-Echo » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #117: 09FIXED in 1 min 6 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/117/ [20:42:35] 03Scap3, 06Discovery, 06Maps: Failed to rollback scap3 deployment - https://phabricator.wikimedia.org/T142792#2555035 (10thcipriani) 05Open>03Resolved [20:43:15] 03Scap3, 06Discovery, 06Maps: Failed to rollback scap3 deployment - https://phabricator.wikimedia.org/T142792#2555041 (10thcipriani) 05Resolved>03Open Landed commit, still unreleased. Leaving open until released. [21:30:31] !log update scap on beta to 3.2.3-1 bugfix release [21:30:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [21:31:48] PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:32:31] 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-MultimediaViewer, 06Reading-Web-Backlog, 13Patch-For-Review, and 4 others: A JSON text must at least contain two octets! (JSON::ParserError) in MultimediaViewer, Echo, Flow, RelatedArticles, MobileFront... - https://phabricator.wikimedia.org/T129483#2555267 [21:33:20] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:34:31] Project beta-scap-eqiad build #115626: 04FAILURE in 92 ms: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/115626/ [21:34:38] PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:35:44] PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:38:46] twentyafterfour: crud. scap package error: ImportError: No module named plugins [21:39:56] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2555319 (10Paladox) Should this be high priority due to it affecting ci nodepool, preventing us from using nodepool until we find and fix mos... [21:42:59] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2555344 (10hashar) Nodepool has a 60 seconds **timeout** for the API requests. If OpenStack replies quickly with a 4xx/5xx it process with the next iteration in... [21:44:32] Project beta-scap-eqiad build #115627: 04STILL FAILING in 89 ms: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/115627/ [21:45:04] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: Nodepool instance creation delay - https://phabricator.wikimedia.org/T143016#2555351 (10Paladox) @hashar we should update nodepool to see if it fixes some of our problems, including patching in some of your changes that still need merging... [21:56:26] Project beta-scap-eqiad build #115628: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/115628/ [21:59:10] Yippee, build fixed! [21:59:10] Project selenium-PageTriage » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #115: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/selenium-PageTriage/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/115/ [22:00:59] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: OpenStack misreports number of instances per project - https://phabricator.wikimedia.org/T143018#2555388 (10hashar) What is the command to get that 32k instances result? The only thing I see approaching is the usage metric: ``` (openstac... [22:06:31] Yippee, build fixed! [22:06:32] Project beta-scap-eqiad build #115629: 09FIXED in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/115629/ [22:08:27] 10Continuous-Integration-Infrastructure, 06Labs, 07Wikimedia-Incident: Investigate upgrade of OpenStack python module for labnodepool1001 - https://phabricator.wikimedia.org/T143013#2555399 (10hashar) Here are roughly the OpenStack related python modules that Nodepool depends on: ``` python-cinderclient... [22:10:26] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Labs, 10Labs-Infrastructure, 15User-greg: Create incident report for CI outage on Aug 10th - https://phabricator.wikimedia.org/T142887#2555420 (10hashar) 05Open>03Resolved a:03greg Published at https://wikitech.wikimedia.org/wik... [22:13:18] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:16:45] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555452 (10hashar) [22:19:01] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555452 (10Paladox) I think this may be fixed in a update to nodepool. I think I saw something that introduces the ability to... [22:20:10] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555477 (10hashar) That is unrelated to Nodepool. Using the openstack CLI client, the instance can not be deleted either :-/ [22:20:50] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555478 (10Paladox) Oh sorry, maybe it is fixed in openstack cli? [22:52:02] 10Beta-Cluster-Infrastructure, 03Scap3 (Scap3-Adoption-Phase1), 10scap, 10Analytics, and 3 others: Set up AQS in Beta - https://phabricator.wikimedia.org/T116206#1743135 (10AlexMonk-WMF) Puppet on this host is broken: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find... [22:54:01] 10Continuous-Integration-Infrastructure, 06Labs, 10Labs-Infrastructure: Delete ci-trusty-wikimedia-278848 instance in contintcloud project - https://phabricator.wikimedia.org/T143058#2555452 (10AlexMonk-WMF) @Paladox, if an instance is stuck in ERROR state that indicates a problem not on the client side :) [23:00:09] 10Beta-Cluster-Infrastructure, 07Puppet: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:00:22] 10Beta-Cluster-Infrastructure, 10Mathoid: Move mathoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142255#2555713 (10AlexMonk-WMF) [23:00:24] 10Beta-Cluster-Infrastructure, 10ContentTranslation-CXserver: Move apertium to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142152#2555714 (10AlexMonk-WMF) [23:00:27] 10Beta-Cluster-Infrastructure, 07Puppet: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:01:12] 10Beta-Cluster-Infrastructure, 10ContentTranslation-CXserver: Move apertium to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142152#2524511 (10AlexMonk-WMF) [23:01:15] 10Beta-Cluster-Infrastructure, 10Mathoid: Move mathoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142255#2528794 (10AlexMonk-WMF) [23:01:17] 10Beta-Cluster-Infrastructure, 07Puppet: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:01:25] 10Beta-Cluster-Infrastructure, 07Puppet: deployment-sca0[12] puppet failure due to issues involving /srv/deployment directory - https://phabricator.wikimedia.org/T143065#2555699 (10AlexMonk-WMF) [23:01:27] 10Beta-Cluster-Infrastructure, 10ContentTranslation-CXserver: Move apertium to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142152#2524511 (10AlexMonk-WMF) [23:01:29] 10Beta-Cluster-Infrastructure, 10Mathoid: Move mathoid to deployment-sca* hosts in Beta Cluster - https://phabricator.wikimedia.org/T142255#2528794 (10AlexMonk-WMF) [23:50:25] (03PS6) 10Awight: Use composer in DonationInterface hhvm tests [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) [23:50:54] (03CR) 10Awight: "ping--simple patch, should be safe and it'll turn a light green for us." [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) (owner: 10Awight) [23:51:56] 10Continuous-Integration-Config, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 03Fundraising Sprint Octopus Untangling, and 2 others: Continuous integration: DonationInterface needs composer variant - https://phabricator.wikimedia.org/T141309#2493985 (10awight)