[06:28:25] Hi, I have a problem to manage my instances [06:28:25] https://wikitech.wikimedia.org/wiki/Special:NovaInstance [06:28:31] does not show any instance [06:29:20] and when I was tring to delete a resource that is no loger needed [06:29:23] i.e. https://wikitech.wikimedia.org/wiki/Nova_Resource:I-000002d5.eqiad.wmflabs [06:29:39] I got the feedback that this resource would not exist [06:40:07] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:54:33] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [06:57:36] PROBLEM - Puppet failure on tools-trusty is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:10:09] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:19:32] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:34] RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [07:38:25] physikerwelt_: typically the solution to problems like yours is a log out and in at wikitech. There's a terrible session bug which I've been unable to stamp out [08:12:04] (03CR) 10Hashar: tox.ini: Specify basepython = python3 (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178571 (owner: 10Legoktm) [08:41:09] andrewbogott_afk: thank you [10:28:55] 3Tool-Labs: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76956#836129 (10Steinsplitter) [10:30:09] 3Tool-Labs: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76956#823567 (10Steinsplitter) [10:30:11] 3Tool-Labs, Wikidata: Wikidata query breaking after DB change (?) - https://phabricator.wikimedia.org/T76699#836132 (10Steinsplitter) [10:31:11] 3Tool-Labs, Wikidata: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#836136 (10Steinsplitter) [10:32:15] 3Tool-Labs, Wikidata: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#836140 (10Steinsplitter) ``` MariaDB [commonswiki_p]> SELECT * FROM user_daily_contribs LIMIT 3; ERROR 2006 (HY000): MySQL server has gone away No connection. Trying to reconnect... Connection id: 1... [11:02:46] andrewbogott: hey! [11:02:54] hello! [11:02:56] andrewbogott: 'wdq-mm-01’ is ‘SHUTOFF’ and wouldn’t restart [11:03:06] Where'd it come from? [11:03:06] I killed a few equivalent instances in the design project before starting this [11:03:27] it’s on virt1008 [11:03:35] I went to sleep, it was working fine, woke up, SHUTOFF, can’t ssh [11:03:42] hm [11:05:37] YuviPanda: better now? [11:05:58] andrewbogott: seems to be. [11:06:02] andrewbogott: did you just restart it from nova? [11:06:07] I just did 'nova start ' [11:06:08] yes. [11:06:17] I don't know why it was shutdown. The rest of the instances on virt1008 look ok. [11:07:25] hmm, ok [11:08:26] there was nothing in the log about it, other than info from when it started up in the first place [11:08:36] hmm [11:08:38] ok [11:08:38] I don't suppose you typed 'shutdown' in the instance? [11:10:00] andrewbogott: no :) [11:10:06] andrewbogott: and restart on the interface didn’t work either [11:10:20] yeah, reboot on the interface is different from 'start' [11:10:30] it was in a state that the interface doesn't expect or handle [11:10:40] ah [11:10:41] I see [11:10:44] hmm, no [11:12:16] andrewbogott: were a bunch more transient failures earlier today, investigating [11:12:41] ok, seem to be the os_version patch introduced yesterday that got fixed quickly [11:13:00] * YuviPanda investigates a bunch more of ‘em [11:14:19] heh, and a sudo-ldap failure [11:17:48] andrewbogott: alright, all of yesterday night’s puppet failures were actual issues, not transient! sudo-ldap and the os_version pathc. [11:17:50] *patch [11:17:55] andrewbogott: the apt-get fix is working so far :) [11:18:11] cool [11:18:21] I have to go -- may be back briefly later tonight. [11:18:47] andrewbogott_afk: cya [14:04:24] https://gerrit.wikimedia.org/r/#/c/178835/ [16:27:49] YuviPanda: I saw you mention elasticsearch and irc logs in the backscroll of -operations. Logstash has an irc input and I use it in this channel to log !log messages into https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/irc. It seems to work pretty well. [16:28:07] bd808: oh, wow. [16:28:23] bd808: that’s nice. Maybe I should just setup an instance with logstash and put all logs into it [16:28:25] and import older logs [16:28:59] That would be pretty cool. [16:30:19] Here's the doc on the irc input -- http://logstash.net/docs/1.4.2/inputs/irc [16:31:27] * bd808 should add that bot to -qa too [16:33:30] bd808: yeah, and we can make it log stuff specifically too [17:36:29] (03CR) 10Legoktm: tox.ini: Specify basepython = python3 (031 comment) [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178571 (owner: 10Legoktm) [18:00:25] Hi Coren! [18:00:38] Hi all! [18:00:49] hi Silke_WMDE [18:01:06] Hi YuviPanda! I'm looking for an information concerning mediawiki + ldap [18:01:28] I'm connecting WMDE's wiki one by one... [18:01:50] Silke_WMDE: aaah, LDAP, I know little about, sadly. #wikimedia-dev or #mediawiki might be able to help better. [18:02:01] or andrewbogott_afk but he’s afk [18:02:10] ah ok YuviPanda, thx! [18:02:15] yw [18:19:08] (03PS1) 10Legoktm: Use transactionPHIDs when getting task transactions [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178874 [18:43:13] (03PS1) 10Legoktm: Add bs4 to requirements.txt [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178879 [18:43:15] (03PS1) 10Legoktm: Don't require an API request to determine the type [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178880 [19:12:38] (03CR) 10Legoktm: [C: 032 V: 032] Use transactionPHIDs when getting task transactions [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178874 (owner: 10Legoktm) [19:12:46] (03CR) 10Legoktm: [C: 032 V: 032] Add bs4 to requirements.txt [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178879 (owner: 10Legoktm) [19:13:00] (03CR) 10Legoktm: [C: 032 V: 032] Don't require an API request to determine the type [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/178880 (owner: 10Legoktm) [19:16:27] !log tools.wikibugs restarting to pick up https://gerrit.wikimedia.org/r/178874 https://gerrit.wikimedia.org/r/178880 [19:16:29] Logged the message, Master [19:18:43] !log tools.wikibugs restarting phab listener for https://gerrit.wikimedia.org/r/178880 [19:18:45] Logged the message, Master [19:43:13] Hi, do we have problem with the labs (my instance ist in "SHUTOFF" stage... don't know why) and I can not reboot it from "wikitech.wikimedia.org" (Failed to reboot instance mwoffliner1.) [19:48:00] YuviPanda|nap: any idea? [19:52:02] PROBLEM - Puppet failure on tools-exec-15 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:52:04] PROBLEM - Puppet failure on tools-exec-gift is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:52:25] PROBLEM - Puppet failure on tools-webgrid-02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:53:35] PROBLEM - Puppet failure on tools-trusty is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:54:27] PROBLEM - Puppet failure on tools-mail is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:55:01] PROBLEM - Puppet failure on tools-exec-13 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [19:55:41] PROBLEM - Puppet failure on tools-webgrid-05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:56:21] PROBLEM - Puppet failure on tools-webgrid-tomcat is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:56:39] PROBLEM - Puppet failure on tools-login is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:56:53] PROBLEM - Puppet failure on tools-exec-12 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:56:55] PROBLEM - Puppet failure on tools-dev is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [19:57:21] PROBLEM - Puppet failure on tools-exec-07 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:58:10] PROBLEM - Puppet failure on tools-webgrid-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:58:42] PROBLEM - Puppet failure on tools-exec-14 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [19:58:48] PROBLEM - Puppet failure on tools-exec-02 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [19:59:52] PROBLEM - Puppet failure on tools-exec-catscan is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:00:04] PROBLEM - Puppet failure on tools-exec-08 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [20:00:04] PROBLEM - Puppet failure on tools-webgrid-03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:00:18] PROBLEM - Puppet failure on tools-shadow is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:01:09] PROBLEM - Puppet failure on tools-exec-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:01:29] PROBLEM - Puppet failure on tools-exec-04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:02:27] PROBLEM - Puppet failure on tools-webproxy is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:03:38] PROBLEM - Puppet failure on tools-webgrid-04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:04:42] PROBLEM - Puppet failure on tools-exec-cyberbot is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:05:49] ... who broke puppet *again*? [20:05:56] PROBLEM - Puppet failure on tools-exec-09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:06:05] PROBLEM - Puppet failure on tools-submit is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:06:11] PROBLEM - Puppet failure on tools-redis is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:07:18] PROBLEM - Puppet failure on tools-exec-03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [20:07:28] PROBLEM - Puppet failure on tools-exec-10 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:07:49] PROBLEM - Puppet failure on tools-exec-05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:08:11] PROBLEM - Puppet failure on tools-exec-06 is CRITICAL: CRITICAL: 25.00% of data above the critical threshold [0.0] [20:09:58] PROBLEM - Puppet failure on tools-exec-11 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:10:32] PROBLEM - Puppet failure on tools-master is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:13:20] PROBLEM - Puppet failure on tools-exec-wmt is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:31:40] RECOVERY - Puppet failure on tools-login is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:30] RECOVERY - Puppet failure on tools-exec-10 is OK: OK: Less than 1.00% above the threshold [0.0] [20:32:49] RECOVERY - Puppet failure on tools-exec-05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:35:41] RECOVERY - Puppet failure on tools-webgrid-05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:36:10] RECOVERY - Puppet failure on tools-redis is OK: OK: Less than 1.00% above the threshold [0.0] [20:37:17] RECOVERY - Puppet failure on tools-exec-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:38:13] RECOVERY - Puppet failure on tools-exec-06 is OK: OK: Less than 1.00% above the threshold [0.0] [20:38:17] RECOVERY - Puppet failure on tools-exec-wmt is OK: OK: Less than 1.00% above the threshold [0.0] [20:39:29] RECOVERY - Puppet failure on tools-mail is OK: OK: Less than 1.00% above the threshold [0.0] [20:40:02] RECOVERY - Puppet failure on tools-exec-11 is OK: OK: Less than 1.00% above the threshold [0.0] [20:40:36] RECOVERY - Puppet failure on tools-master is OK: OK: Less than 1.00% above the threshold [0.0] [20:40:52] RECOVERY - Puppet failure on tools-exec-09 is OK: OK: Less than 1.00% above the threshold [0.0] [20:41:20] RECOVERY - Puppet failure on tools-webgrid-tomcat is OK: OK: Less than 1.00% above the threshold [0.0] [20:42:01] RECOVERY - Puppet failure on tools-exec-15 is OK: OK: Less than 1.00% above the threshold [0.0] [20:42:05] RECOVERY - Puppet failure on tools-exec-gift is OK: OK: Less than 1.00% above the threshold [0.0] [20:42:25] RECOVERY - Puppet failure on tools-webgrid-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:43:34] RECOVERY - Puppet failure on tools-trusty is OK: OK: Less than 1.00% above the threshold [0.0] [20:45:02] RECOVERY - Puppet failure on tools-exec-13 is OK: OK: Less than 1.00% above the threshold [0.0] [20:45:07] RECOVERY - Puppet failure on tools-webgrid-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:46:54] RECOVERY - Puppet failure on tools-exec-12 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:40] RECOVERY - Puppet failure on tools-webgrid-04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:48:52] RECOVERY - Puppet failure on tools-exec-02 is OK: OK: Less than 1.00% above the threshold [0.0] [20:49:42] RECOVERY - Puppet failure on tools-exec-cyberbot is OK: OK: Less than 1.00% above the threshold [0.0] [20:49:52] RECOVERY - Puppet failure on tools-exec-catscan is OK: OK: Less than 1.00% above the threshold [0.0] [20:50:16] RECOVERY - Puppet failure on tools-shadow is OK: OK: Less than 1.00% above the threshold [0.0] [20:51:06] RECOVERY - Puppet failure on tools-submit is OK: OK: Less than 1.00% above the threshold [0.0] [20:51:06] RECOVERY - Puppet failure on tools-exec-01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:51:30] RECOVERY - Puppet failure on tools-exec-04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:51:57] RECOVERY - Puppet failure on tools-dev is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:27] RECOVERY - Puppet failure on tools-webproxy is OK: OK: Less than 1.00% above the threshold [0.0] [21:02:18] RECOVERY - Puppet failure on tools-exec-07 is OK: OK: Less than 1.00% above the threshold [0.0] [21:03:09] RECOVERY - Puppet failure on tools-webgrid-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:03:41] RECOVERY - Puppet failure on tools-exec-14 is OK: OK: Less than 1.00% above the threshold [0.0] [21:05:01] RECOVERY - Puppet failure on tools-exec-08 is OK: OK: Less than 1.00% above the threshold [0.0]