[00:34:25] 3Multimedia, Wikimedia-Labs-Other: Migrate our labs instance(s) to eqiad - https://phabricator.wikimedia.org/T77255#832560 (10scfc) [01:04:13] andrewbogott: can you tell me how to look up who created an instance? [01:04:24] Coren: ^ [01:04:25] YuviPanda: on virt1000: [01:04:38] $ source ~root/novaenv.sh [01:04:53] $ nova --os-tenant-name list [01:05:04] (Grab the id of the instance of interest) [01:05:09] $ nova show [01:06:13] andrewbogott: hmm, it shows me when it was created, but not who? [01:06:30] user_id is just novaadmin [01:06:43] Hm, try for a recently-created instance [01:06:51] I think 'novadmin' means that it's an ancient instance. [01:06:54] aaah [01:07:00] this is an ancient insatnce [01:07:08] so I guess we can’t for these. [01:07:21] Or I'm wrong and that's not how you check… let me try [01:07:33] ok [01:08:26] yeah, sure enough for a new instance it shows "andrew" as the user_id [01:08:45] I bet that instances which I migrated from tampa via a script show as 'novaadmin' [01:09:16] Sorry that isn't very helpful :( [01:09:17] andrewbogott: yeah, that makes sense [01:09:20] ’tis ok :) [01:09:35] andrewbogott: I just wanted a new instance, and was trying to get the wikidata people to delete one so I can :) [01:09:48] but I’ll see if I can kill one of my other instances from somewhere else instead. [01:10:00] andrewbogott: I suspect the new machines won’t be online this week? [01:10:23] I don't know. It should be pretty trivial to get them up but we're still fighting with HP bios and such. [01:10:31] andrewbogott: right. [01:10:45] I thought faidon was about to coach me but he must've gone to sleep [01:11:17] andrewbogott: ah, heh :) [01:11:28] YuviPanda: if you happen to be a partman expert... [01:11:32] andrewbogott: also, do you think you can review a debianization I just did? first time I’m doing this from scratch. [01:11:34] andrewbogott: https://bitbucket.org/yuvipanda/wikidataquery/pull-request/1/wip-initial-debianization [01:12:00] andrewbogott: heh, thankfully never used it :) I’ve bad luck with partitioning - last time I used fdisk/gparted I managed to lose a lot of data [01:12:08] but that was what, 6 years ago? [01:13:10] The debian stuff looks right to me at first glance, but I can't predict if it'll work or not. [01:13:15] If it works, it's probably right :) [01:13:33] andrewbogott: it does work! [01:13:56] Most of the debian packages I've built were using automatic tools to generate all the package files. I haven't ever written them from scratch. [01:14:04] andrewbogott: oh, this also is built off dh_make [01:14:19] Yah, probably trustworthy then [01:14:20] andrewbogott: but my previous ones were built off python-stdeb, which is pretty much a ‘one command’ type of thing. [01:14:27] andrewbogott: cool :) [01:14:34] andrewbogott: is /usr/share a good place to put static html files? [01:15:17] Seems ok -- I've seen them all over the place. [01:15:31] godog would be good to ask about this stuff, but he's most likely sleeping. [01:16:07] andrewbogott: yeah, will ask him tomorrow [01:16:10] andrewbogott: thanks! [01:16:34] andrewbogott: heh, I think you’ve ‘gone to sleep’ three times since I woke up today? [01:16:38] andrewbogott: this TZ is going to be interesting :) [01:16:54] I woke up in the night for the call. So that's one extra bedtime... [01:17:01] Also napping extensively due to jetlag. [01:17:09] Waking up this morning felt pretty good though, maybe I'm cured [01:17:51] heh, wait till 4pm [01:17:54] that’s when it usually hits me [01:18:06] at least on eastward trips [01:18:18] Coren: can I get a bit of partman advice if you are still around? [01:33:40] !log design deleted redcushion and tea-rex as unused after checking with prtksxna [01:33:43] Logged the message, Master [02:37:03] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:40:37] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:40:47] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:41:43] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:42:09] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:42:17] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:42:49] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:43:08] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:43:31] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:44:27] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:44:29] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:44:48] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:48:10] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:48:54] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:50:10] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0] [02:50:18] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:51:16] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:51:32] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:51:58] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:52:28] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:53:00] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:53:20] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:53:42] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [02:54:27] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:55:04] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [02:55:10] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:55:20] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:55:36] PROBLEM - Puppet failure on tools-trusty is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:56:37] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [02:58:00] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [02:59:40] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:59:52] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:59:56] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [03:01:10] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 75.00% of data above the critical threshold [0.0] [03:09:39] I am… working on those [03:32:18] Hi I have problem with login to https://gerrit.wikimedia.org what should i do? [03:32:36] is it possible to reset the password? [03:40:54] reza1615: I believe that your gerrit login/password are the same as for wikitech. [03:41:04] So if you're unable to log in either place, you can reset your password on wikitech. [03:41:05] thanks [03:41:10] now it is ok [03:41:12] :) [03:42:14] but when i type become rezabot [03:42:24] it shows this error [03:42:30] sudo: sorry, a password is required to run sudo [03:42:45] I logout and login [03:46:35] sudo may be temporarily broken -- I'm working on that right now. [03:46:55] I check it now [03:47:36] it shows the same erorr [03:47:45] error [04:06:46] 3Tool-Labs-tools-Other: SVGCheck gives "Not an SVG" error in both IE and Chrome - https://phabricator.wikimedia.org/T75470#832818 (10Trlkly) Okay, I tracked down the issue. It seems that an installation of Inkscape set the mimetype of SVG files to "application/svg" in the registry. Changing it to "image/svg+xml"... [04:07:54] andrewbogott: for https://wikitech.wikimedia.org/wiki/Help:Tool_Labs#I.27m_being_prompted_for_a_password_when_I_try_to_.27become_my-tool-account.27._What.27s_wrong.3F [04:08:06] reza1615: as I said, I am currently working on that [04:08:17] ok thanks [04:11:45] 3Wikimedia-Labs-Infrastructure: Jenkins fails to pass cxserver-npm tests - https://phabricator.wikimedia.org/T77958 (10KartikMistry) 3NEW p:3Triage [04:25:30] 3Wikimedia-Labs-Infrastructure: Jenkins fails to pass cxserver-npm tests - https://phabricator.wikimedia.org/T77958#832833 (10KartikMistry) 5Open>3Invalid a:3KartikMistry OK. This was the actual failure. Fixed in, https://gerrit.wikimedia.org/r/#/c/177745/4 [04:26:00] 3Wikimedia-Labs-Infrastructure: Jenkins fails to pass cxserver-npm tests - https://phabricator.wikimedia.org/T77958#832836 (10KartikMistry) 5Invalid>3Resolved a:5KartikMistry>3None [04:32:57] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [04:34:19] reza1615: can you try logging out and in again, and then trying 'become' again? [04:34:28] btw, is this a tool that you've become w/out errors in the past? [04:34:36] yes now i will do [04:35:10] andrewbogott: thanks now it works [04:35:12] :) [04:35:15] great! [04:36:34] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [04:38:42] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [04:39:26] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:40:04] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [04:40:08] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [04:40:14] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [04:40:34] RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [04:41:17] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [04:41:39] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [04:41:55] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [04:42:29] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [04:43:01] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [04:43:21] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [04:44:40] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [04:44:54] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [04:44:55] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [04:45:23] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [04:45:49] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:46:09] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:47:03] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [04:47:19] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [04:47:49] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [04:48:11] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [04:48:29] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:49:27] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [04:50:21] For those following along at home… all of those shinken messages are legit. I sent an email to labs-l with details. [04:50:39] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:51:41] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [04:52:08] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [04:53:12] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [04:53:55] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [04:54:29] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [04:54:49] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [04:55:19] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [06:52:58] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:06:14] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:09:40] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:10:24] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:11:02] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:11:08] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:11:36] PROBLEM - Puppet failure on tools-trusty is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:12:16] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:12:33] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:13:29] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:13:57] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:14:20] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:15:40] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:15:54] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:15:56] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:16:22] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:16:50] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:17:12] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:17:40] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:18:00] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:04] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:18:18] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [07:18:49] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:18:59] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [07:19:29] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:21:37] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:22:39] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:23:08] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:24:06] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:25:27] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:25:29] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:25:48] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:29:09] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:36:10] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:16] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:31] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:21] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:41] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:40:25] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:01] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:07] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [07:41:35] RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [07:42:39] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [07:43:29] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [07:43:59] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:00] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:13] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:43] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:53] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [07:45:53] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [07:46:21] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [07:46:39] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:46:51] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:47:09] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:04] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:10] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:20] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [07:48:50] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [07:49:10] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [07:49:30] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:50:26] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [07:50:28] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [07:52:43] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [07:55:50] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:57:35] 3Tool-Labs: Set up redirect webserver for toolserver.org - https://phabricator.wikimedia.org/T62238#832958 (10silke) Hi Coren and all, could we please get a status update? This needs to be done. Now. [09:01:12] andrewbogott: am glad they were all legit :) [09:01:43] yep. Noisy but helpful. [09:02:38] andrewbogott: yeah. I need to fix our notification setups to batch them [09:16:05] GerardM-: hi! [09:16:12] Hoi [09:16:19] I heard the good news from Magnus [09:16:28] GerardM-: oh? [09:16:39] what did you hear? [09:16:50] you working with him of puppetising WDQ [09:17:46] GerardM-: ah, yes. I’m about 70% done, actually :) [09:18:01] need him to fix https://bitbucket.org/magnusmanske/wikidataquery/issue/11/make-path-to-html-css-static-assets to be fully done, but that’s fairly trivial [09:18:45] when you are done, may I ask you some questions so that I can write a proper blogpost about it ? [09:19:15] GerardM-: sure [09:20:14] godog: hey! First time I’m packaging a non python package, can you take a look to see if I’m doing something wrong? https://bitbucket.org/magnusmanske/wikidataquery/pull-request/3/wip-debian-packaging/diff [09:23:08] hey YuviPanda, sure I'll take a look in a few (likely 12-13 UTC) [09:23:15] godog: sure! thanks [09:24:22] GerardM-: we’ve also discovered other areas where we can trivially improve performance :) [09:24:45] tell me more [09:25:12] GerardM-: oh, nothing much. it was using NFS instead of local instance storage for some files, and moving would provide a perf boost [09:26:13] like nothing much ... probably similar improvements as moving to this Facebook PHP thingie [09:27:03] !log wdq-mm Add Magnus Manske to project and set as projectadmin [09:27:08] Logged the message, Master [09:57:38] 3Tool-Labs-tools-Other: Migrate http://toolserver.org/~purodha/sample/dbswithuser.php to Tool Labs - https://phabricator.wikimedia.org/T63028#833087 (10TTO) "I do [plan to migrate]. I only do not have plans as to when I have time to do it. --Purodha Blissenbach (talk) 19:39, 8 December 2014 (UTC)" [10:03:40] PROBLEM - Free space - all mounts on tools-webproxy is CRITICAL: CRITICAL: tools.tools-webproxy.diskspace._var.byte_percentfree.value (<11.11%) [10:13:41] RECOVERY - Free space - all mounts on tools-webproxy is OK: OK: All targets OK [10:25:53] YuviPanda: left a few comments on the pull request [10:28:37] godog: thank you very much! [10:28:54] godog: re: why not wikidata-query, that’s because in production people are looking at graph databases to build a wikidata-query [10:28:59] godog: and I don’t want any potential conflicts [10:29:15] godog: and hmm, I’m not sure if this will actually ever run on prod. at the moment it’s running in a screen in one of magnus’ instances. [10:32:57] YuviPanda: ah okay, good packaging exercise then [13:10:01] 3Tool-Labs, Labs-Team: Labs proxy + tools proxy should use ssl_ciphersuite - https://phabricator.wikimedia.org/T77987 (10yuvipanda) 3NEW p:3Triage [13:10:33] 3Beta-Cluster, Labs-Team: Core dumps fill up /var on labs instances - https://phabricator.wikimedia.org/T1259#833362 (10yuvipanda) [13:53:52] !log wdq-mm created instance wdq-mm-test-1 [13:53:57] Logged the message, Master [14:27:35] andrewbogott_afk: Hey there. If you have a minute today I have a bizzare issue: a new instance of mine has its public IP unreachable for some reason. [14:28:24] Coren: is this the toolserver.org redirect? [14:28:32] YuviPanda: Yeah. [14:28:43] Coren: ah, ok! :) Are you puppetizing it? [14:29:05] I will, partly. But the actual data is PII and so can't be in puppet. [14:29:41] actual data? as in the redirects themselves? [14:29:50] Mostly user email [14:30:14] The web redirects should be okay. [14:30:45] But those I intend to not be modifyable. [14:31:06] aaah [14:31:06] ok [14:32:39] 3Phabricator, Wikimedia-Labs-Other: phab-01.wmflabs.org test instance's statuses are out of date - https://phabricator.wikimedia.org/T76943#833518 (10Aklapper) Hmm, I'm not admin on that one plus not sure if we ever pull the updated stuff in Puppet... [14:36:53] Coren: hmm, if I wanted to copy data from NFS of one project to another, which host should I get into? [14:37:07] labstore1001. Look in /srv [14:57:49] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:59:06] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:59:10] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:59:18] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:00:28] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:01:26] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:02:38] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:03:40] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:04:53] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:05:07] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:05:11] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:06:17] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:06:29] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:06:51] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:07:09] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:08:33] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:08:59] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:24:03] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:27:40] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:27:48] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [15:28:42] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:06] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [15:29:18] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:08] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [15:30:28] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [15:31:26] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [15:31:30] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [15:31:50] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:56] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [15:35:12] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [15:36:15] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:37:13] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:31] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [15:38:55] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:50] 3Tool-Labs: Tools labs exec host issue - https://phabricator.wikimedia.org/T78010 (10Magnus) 3NEW p:3Triage [16:53:33] Coren: toolsdb is in a prod machine right? do you know which one? [16:53:57] I should hope so, given that I set it up. :-) labsdb1003 [16:54:05] ah, ok :) [16:54:27] Coren: hmm, then what’s c3 on? they’ve different IPs in /etc/hosts [16:54:57] Oh, hum Typo. 1004 [16:55:06] aah, cool :) [16:55:07] ty [16:55:08] Dang, was there another puppet storm after I logged out? [16:55:49] andrewbogott: yup, a statsd related change in diamond module, godog reverted [16:55:54] Coren: which instance and IP are failing? [16:55:57] tools-db was 1005 in the past if this hasn't changes the last week [16:56:06] YuviPanda: was that another one of the patches that I merged? [16:56:33] andrewbogott: have you merged any patches in diamond recently? [16:56:40] (Wondering if I have a perfect streak, since I caused the earlier ones) [16:56:50] I don't think so, so probably not me :) [16:56:56] Merlissimo: indeed, you are correct :) [16:57:38] andrewbogott: relic.wmflabs.org (208.80.155.197) -> relic in too;lserver-legacy [16:57:59] YuviPanda: btw do you know how long magnus wdq will be down? i have seen that you last modifiend something there [16:58:10] YuviPanda: Bah. Let's pretend I actually know what I'm talking about okay? :-) [16:58:39] Coren: :D [16:58:39] Merlissimo: it’s down now? [16:58:49] Merlissimo: seems to be. let me restart [16:58:51] wrong gateways 502 [16:59:05] or bad gateway [16:59:11] Merlissimo: restarting again [16:59:48] marktraceur: I’ve restarted it now [17:00:11] Thanks. [17:00:14] Restarted what? [17:00:32] gah [17:00:34] I meant Merlissimo [17:00:37] :) [17:00:39] YuviPanda: thx, seems to be up again [17:00:56] Merlissimo: cool. I’m puppetizing it, should be better once that’s done [17:03:02] andrewbogott: on the plus side, no apt-get related errors yet. [17:03:18] andrewbogott: but maybe that’s been lost in the storms, but I guess not. [17:03:20] YuviPanda: well, we've barely given it the chance :) [17:03:32] andrewbogott: hehe :) yeah, I’ll give it a week. [17:04:42] Coren: I'm going to just release and reassing the IP, just in case that helps. Nova certainly doesn't know about the IP at the moment, even though it shows up on wikitech [17:05:05] I tried that, but from Wikitech. [17:05:40] Oh, actually -- you need to click on 'associate' [17:05:46] It has an IP and DNS but isn't pointed anywhere [17:08:29] Coren: better? [17:18:55] andrewbogott: Yep. Thankee. But I /did/ 'associate' [17:19:22] Coren: that's weird then :( We'll see if it sticks this time... [17:26:40] andrewbogott: I’ve had this happen also, I suspect nova auth issue not being reflected on wikitech [17:26:44] might be totally wrong [17:28:19] YuviPanda: and you saw the culumn filled in on the web page? [17:28:43] andrewbogott: yeah, this was for adding valhallasw`cloud to a project as admin. [17:28:45] didn’t really take [17:31:08] Does this make sense as a "we don't have a redirect" landing page? http://relic.wmflabs.org/~foo/ [17:32:15] Coren: I like it. [17:33:16] Coren: I kind of think the image should link to the index of tools [17:33:47] Hm. I was hesitating between the docs (to explain) and the index. [17:33:56] And maybe a page title like 'tool not found' or something, to make more explicit that that's an error page [17:34:22] Certainly the /second/ time anyone sees that page they'll want the index. [17:34:31] But I can see the merits either way [17:35:23] There's a link to the index; but since there's also a link to the docs in the text I suppose the "bigger" link should reasonably be the more immediately useful one. [17:35:46] Both changes made. [17:36:48] The not-an-error (just /) still points to the doc since anyone reaching that wasn't looking for a specific tool but following a link to the TS [17:39:33] yep, looks great. [19:14:55] (03PS1) 10Legoktm: Fix regexes for #wikimedia-labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178561 [19:19:31] (03PS1) 10Legoktm: Send User-Bd808-Test to /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178563 [19:20:50] (03PS1) 10Legoktm: Fix flake8 errors [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178564 [19:21:04] (03CR) 10Legoktm: [C: 032 V: 032] Fix flake8 errors [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178564 (owner: 10Legoktm) [19:21:38] (03PS2) 10Legoktm: Fix regexes for #wikimedia-labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178561 [19:21:44] (03CR) 10Legoktm: [C: 032 V: 032] Fix regexes for #wikimedia-labs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178561 (owner: 10Legoktm) [19:21:58] (03PS2) 10Legoktm: Send User-Bd808-Test to /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178563 [19:22:03] (03CR) 10Legoktm: [C: 032 V: 032] Send User-Bd808-Test to /dev/null [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178563 (owner: 10Legoktm) [19:22:44] !log tools.wikibugs restarting for https://gerrit.wikimedia.org/r/178561 https://gerrit.wikimedia.org/r/178563 [19:22:49] Logged the message, Master [19:34:38] (03PS1) 10Legoktm: tox.ini: Specify basepython = python3 [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178571 [19:45:43] (03PS1) 10Legoktm: Revert "Fix regexes for #wikimedia-labs" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178578 [19:45:48] (03CR) 10Legoktm: [C: 032 V: 032] Revert "Fix regexes for #wikimedia-labs" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178578 (owner: 10Legoktm) [19:46:51] !log tools.wikibugs restarting for https://gerrit.wikimedia.org/r/178578 [19:46:53] Logged the message, Master [22:13:06] 3Tool-Labs: Make archive table partially accessible on Wikimedia Labs - https://phabricator.wikimedia.org/T51088#835068 (10Ricordisamoa)