[07:13:40] 10Traffic, 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3930185 (10demon) Is there anything left here, now that everything in the summary is done? [13:48:24] 10Traffic, 10Operations, 10PAWS, 10Pywikibot-Commons, and 2 others: Server error (500) while trying to download files from Commons from PAWS - https://phabricator.wikimedia.org/T178567#3930577 (10Chicocvenancio) 05Open>03Resolved a:03Chicocvenancio Creating a new task to deal with pywikibot's side. [17:40:43] 10netops, 10Operations: reconfigure esams switch port for new bastion - https://phabricator.wikimedia.org/T186021#3931220 (10Dzahn) p:05Triage>03High [17:42:14] 10netops, 10Operations: reconfigure esams switch port for new bastion - https://phabricator.wikimedia.org/T186021#3931220 (10Dzahn) [18:08:45] 10Domains, 10Traffic, 10Operations, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3931431 (10Dzahn) 05Open>03stalled We need some more information what exactly is requested, please. Is the expectation a DNS... [18:29:05] mutante: about T186021, is the host online? I can find its mgmt interface, but not its prod interface [18:29:05] T186021: reconfigure esams switch port for new bastion - https://phabricator.wikimedia.org/T186021 [18:29:59] XioNoX: i think i turned it off again, should i power it up? didnt want it to boot to the old OS [18:30:23] it had already been decomed but was before the wipe step [18:31:12] mutante: just briefly should do it, there is no obvious matching interface description, so the only two ways to find its switch port is either trace the cable, or see where its mac shows up [18:31:32] ok [18:32:51] !log powercyling amslvs4, to be reinstalled as bast3003 [18:33:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:34:17] mutante: if you have access to it's management console you can avoit it booting into the old system from the menu [18:34:24] mutante: a4:ba:db:38:e4:cb is the mac that should show up, right? [18:34:29] right now i see: [18:34:34] Defauafor Remote1Access1Setup within 5 sec...... [18:34:34] Press [18:35:49] XioNoX: yes, the MAC is correct [18:36:00] first interface that i got with raacadm getsysninfo [18:37:34] volans: yea, i'm hitting F12 to PXE boot [18:37:43] :) [18:37:44] here we go, it comes up and should attempt DHCP now [18:38:18] it also shows (MAC Address - A4BADB38E4CB) during boot fwiw [18:39:05] mutante: It doesn't show up. Is it possible that its switch port got disabled in the past? [18:39:13] like decom' or something? [18:39:19] it doesnt go past: [18:39:24] Defauafor Remote1Access1Setup within 5 sec...... [18:39:28] Press [18:39:37] yes, that is possible [18:39:44] it was decomed before this [18:39:51] i am just supposed to reactivate one [18:40:16] XioNoX: yea, probably it's entirely disabled [18:40:24] mutante: ah, let's find its decom task then. maybe it's not even plugged anymore [18:40:24] hrmm [18:40:49] https://phabricator.wikimedia.org/T87790 [18:41:03] "amslvs1 and amslvs2 i'm now disconnecting as they're in the way." [18:41:04] Please cancel your "Docusign" signing request if you have one in progress as Arzhel is not an authorized signer. [18:41:06] but not 4! [18:41:13] er [18:42:28] mutante: yeah, do you find a switch port information in the task? [18:44:42] XioNoX: no, there is none. i created that ticket back in 2015 but i dont have switch access [18:44:52] mutante: or maybe the task where they got set up? [18:47:28] mutante: let's try something, is the host still powered on? [18:49:28] XioNoX: i searched all old tickets. the only reference to amslvs that isnt about decoming them: [18:49:33] amslvs2 has a bad drive 5 years ago in RT [18:49:44] XioNoX: yes, it's powered on [18:53:40] mutante: I tried to enable the deactivated ports of the switch in the same rack to see if the mac shows up, but no luck, it might be connected to a different switch member [18:54:34] XioNoX: hmm.. thanks for trying. maybe special because it used to be an LVS [18:55:11] I don't want to blindly test disabled ports too much though as it might have side effects [18:55:15] when reconnecting to console i didnt see output anymore [18:55:21] maybe it _did_ shutdown again.. grm [18:55:35] powercycles one more time [18:56:38] XioNoX: i understand. not sure what we can do then without physical esams access [18:57:00] the whole point was that it was supposed to be easier than waiting for hardware refresh though :p [18:57:19] a temp solution until the new hardware arrives basically [18:57:19] mutante: physically tracing the cable is the next step [18:57:49] mutante: once we have netbox in production, we can have all this information there :) [18:58:50] XioNoX: yep, so i should set i to ops-esams then .. YAY @ netbox! :) [18:59:47] how CRIT is it if we lost the European bastion [18:59:59] it is currently on a single disk afaict [19:00:13] but of course we have 3 other bastions.. just not as close for European users [19:03:51] mutante: the bastion functionality is not a big deal IMHO, but IIRC it does other stuff too right? (prometheus comes to mind) [19:06:01] volans: true, and that part makes me sigh a bit. because i remember how bastions did a lot more stuff and then we removed it and finally ganglia as well and always disliked that bastion isnt _just_ bastion [19:06:16] yeah [19:06:24] it's also installserver::tftp in caching-dc's [19:06:35] because we want to minimize the number of hosts.. understandably [19:06:35] the installserver and ipmi part doesn't worry me too much [19:06:48] as long as we can point them to the main DCs ones [19:06:59] but lossing metrics would be bad, yes [19:07:02] true [19:07:29] there is a task somewhere about "misc" servers in caching POPs [19:08:14] yeah, we chatted about it few days ago too, I think we're not yet still clear on the long term solution [19:08:34] a small ganeti cluster was propsed too IIRC [19:43:25] 10netops, 10Operations, 10ops-esams: reconfigure esams switch port for new bastion - https://phabricator.wikimedia.org/T186021#3931926 (10Dzahn) [19:46:02] 10netops, 10Operations, 10ops-esams: reconfigure esams switch port for new bastion - https://phabricator.wikimedia.org/T186021#3931941 (10Dzahn) 05Open>03stalled We don't know the switch port this was connected to and it's already disabled. So it wasn't possible for ayounsi to do this from remote. Also... [19:46:08] 10netops, 10Operations, 10ops-esams: reconfigure esams switch port for new bastion - https://phabricator.wikimedia.org/T186021#3931946 (10Dzahn) a:05Dzahn>03None [19:47:41] 10Domains, 10Traffic, 10Operations, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3911827 (10Bawolff) Have we considered just having it as a subdirectory of https://doc.wikimedia.org/ ? It seems like a documentat... [19:52:52] 10Domains, 10Traffic, 10Operations, 10WMF-Design, and 2 others: Create subdomain for Design and Wikimedia User Interface Style Guide - https://phabricator.wikimedia.org/T185282#3931973 (10Volker_E) @Bawolff There have been conversations in the Design team around the location. The Style Guide is planned as... [21:00:39] 10Traffic, 10Operations, 10Performance-Team (Radar): load.php response taking 160s (of which only 0.031s in Apache) - https://phabricator.wikimedia.org/T181315#3932349 (10Imarlier)