[00:26:33] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[01:06:37] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[01:46:42] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[02:26:47] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[03:06:52] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[03:46:57] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[04:27:02] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[05:07:06] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[05:47:11] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[06:25:19] <icinga-wm>	 PROBLEM - ORES web node labs ores-web-05 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:25:29] <icinga-wm>	 PROBLEM - ORES web node labs ores-web-03 on ores.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:27:15] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[07:07:20] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[07:47:25] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[08:27:30] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[09:07:35] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[09:27:58] <icinga-wm>	 PROBLEM - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/ores is alerting: 5xx rate (Change prop) alert.
[09:46:05] <icinga-wm>	 RECOVERY - https://grafana.wikimedia.org/dashboard/db/ores grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/ores is not alerting.
[09:47:40] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[10:27:44] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[11:07:49] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[11:47:54] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[12:27:58] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[12:38:31] <travis-ci>	 eisenhaus335/wikilabels#34 (master - ea63740 : eisenhaus335): The build was broken. https://travis-ci.org/eisenhaus335/wikilabels/builds/330337080
[12:40:08] <travis-ci>	 eisenhaus335/wikilabels#35 (master - cdd5587 : eisenhaus335): The build was broken. https://travis-ci.org/eisenhaus335/wikilabels/builds/330337754
[12:40:34] <travis-ci>	 eisenhaus335/wikilabels#36 (master - 488febb : eisenhaus335): The build was broken. https://travis-ci.org/eisenhaus335/wikilabels/builds/330337931
[13:08:03] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[13:12:28] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10ORES, 10Release-Engineering-Team (Kanban), 10User-zeljkofilipin: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3909537 (10zeljkofilipin) a:05zeljkofilipin>03None
[13:48:03] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[14:12:13] <travis-ci>	 eisenhaus335/wikilabels#45 (master - df038ab : eisenhaus335): The build was fixed. https://travis-ci.org/eisenhaus335/wikilabels/builds/330371523
[14:19:58] <Zppix>	 I have a pdf of an awesome Ai-related book if anyone wants it let me know
[14:20:21] <apergos>	 what's the book? is it "ai" or "ml" or...?  and.... is there some maths in it? 
[14:20:24] * apergos asks hopefully
[14:21:50] <Zppix>	  apergos its deep learning by ian goodfellow
[14:21:57] <apergos>	 ah. have, but thanks
[14:22:05] <apergos>	 and it's a very nice text indeed
[14:22:13] <Zppix>	 I only read a bit
[14:28:08] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[14:29:14] <Zppix>	 If paladox was around...
[14:40:43] <awight>	 back in 10.
[15:08:13] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[15:08:35] * halfak is "AFK" but peeks in. 
[15:10:37] <awight>	 lol
[15:11:03] <awight>	 I got fired up about this, done now: https://phabricator.wikimedia.org/T185116
[15:11:35] <awight>	 Currently staring at a wikilabels failure under vagrant
[15:39:14] <awight>	 halfak: https://github.com/wiki-ai/editquality/pull/115 is corrected, if it’s that kind of "AFK"
[15:44:13] <awight>	 :)
[15:44:48] <wikibugs>	 10Scoring-platform-team (Current), 10MediaWiki-Vagrant, 10MediaWiki-extensions-ORES, 10ORES, and 2 others: ORES MediaWiki-Vagrant roles should be ported to Stretch - https://phabricator.wikimedia.org/T184077#3909897 (10awight) I'm stuck at a strange error, it seems that the wikilabels database is created b...
[15:48:17] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[16:28:17] <icinga2-wm>	 PROBLEM - Host ORES-web05.experimental is DOWN: check_ping: Invalid hostname/address - ores-web-05.ores.eqiad.wmflabsUsage:check_ping -H <host_address> -w <wrta>,<wpl>% -c <crta>,<cpl>% [-p packets] [-t timeout] [-4
[16:35:09] <awight>	 Amir1: I’m confused by campaign_id_seq.  It doesn’t seem to be a real sequence?
[16:54:33] <awight>	 Amir1: no rush, https://github.com/wiki-ai/wikilabels/pull/224
[16:55:11] <awight>	 Amir1: Any idea how to diagnose high CPU usage by the wikilabels service?  Is it debug logging?
[16:55:51] <awight>	 ores-wsgi is killing the CPU also, but I don’t see anything in the logs!
[16:57:42] <Amir1>	 I'm not sure
[16:58:26] <Amir1>	 awight: the travis is failing on it
[16:58:44] <awight>	 ok thanks for the note
[17:04:25] <Zppix>	 halfak: icinga done
[17:05:04] <halfak>	 awight, why do you think it's not a real sequence?
[17:05:12] <halfak>	 Zppix, thanks :) 
[17:05:41] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10VPS-project-icinga2: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148#3907580 (10Halfak) a:03Halfak
[17:05:52] <Zppix>	 Np
[17:06:00] <awight>	 halfak: It’s not declared anywhere in the schema or db migrations, nor used from code.  But now I see that “setval” is indeed applied to sequences.
[17:06:08] <awight>	 Do you know any background?
[17:06:28] <icinga2-wm>	 CUSTOM - Host ORES-worker01.experimental is UP: PING OK - Packet loss = 0%, RTA = 0.84 ms zppix Test notification
[17:06:37] <Zppix>	 Yay
[17:07:11] <halfak>	 hmm.  awight might have been due to an issue with importing data and updating the sequence safely. 
[17:07:18] <halfak>	 Not sure why we only do it with campaign ID. 
[17:07:37] <awight>	 Instead of using nextval() on an explicit sequence, we’re setting one sequence from another.  I believe that postgres creates an implicit sequence on the auto increment column.
[17:08:40] <awight>	 Yeah, check out section 8.1.4, https://www.postgresql.org/docs/9.1/static/datatype-numeric.html
[17:09:13] <awight>	 So campaign_id_seq is implicitly created, and we shouldn’t have to nurse it with setval...
[17:09:25] * awight looks at the test failure
[17:12:59] <awight>	 Amir1: Note for later, https://scrutinizer-ci.com/g/wikimedia/mediawiki-extensions-ORES/inspections phpcs-run has been consistently failing to run.
[17:13:55] <Amir1>	 Thanks, noted, will work on it soon
[17:18:18] <awight>	 halfak: I think you’re right, we’re somehow importing data in a way that doesn’t increment the ID.
[17:19:38] <halfak>	 Maybe we should just have the sequence be set at the end of our imports and not in our database updating code. 
[17:20:04] <awight>	 +1, or there’s something wrong with the import, like directly specifying .id
[17:20:41] <awight>	 yup.
[17:42:02] <awight>	 There’s the correct fix ^
[17:47:27] <awight>	 (undid my push to master)
[17:49:54] <wikibugs>	 10Scoring-platform-team (Current), 10MediaWiki-Vagrant, 10MediaWiki-extensions-ORES, 10ORES, and 2 others: ORES MediaWiki-Vagrant roles should be ported to Stretch - https://phabricator.wikimedia.org/T184077#3910400 (10awight) The wikilabels thing was a code issue, a fix is ready for review.  I'm still see...
[17:50:53] <travis-ci>	 wiki-ai/wikilabels#294 (no_seq - f402932 : Adam Roses Wight): The build was fixed. https://travis-ci.org/wiki-ai/wikilabels/builds/330468032
[17:51:52] <wikibugs>	 (03CR) 10Awight: [C: 04-1] "Two things to double-check" (032 comments) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/404886 (https://phabricator.wikimedia.org/T182799) (owner: 10Halfak)
[18:01:50] <awight>	 Amir1: I removed phpcs from Scrutinizer, it’s redundant with Jenkins tests anyways.
[18:11:54] <awight>	 Amir1: and it runs again.
[19:23:53] <wikibugs>	 10Scoring-platform-team, 10JADE: Design "rationales" integration for JADE feedback - https://phabricator.wikimedia.org/T185247#3910813 (10awight)
[19:28:20] <wikibugs>	 10Scoring-platform-team (Current), 10editquality-modeling, 10User-Ladsgroup, 10artificial-intelligence: Implement code generation for model makefile maintenance - https://phabricator.wikimedia.org/T168455#3910860 (10awight)
[19:44:04] <wikibugs>	 10Scoring-platform-team, 10Collaboration-Community-Engagement, 10MediaWiki-extensions-ORES, 10Patch-For-Review, 10User-notice-collaboration: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#3910891 (10awight) @Catrope This should be unblocked from our perspective.  The...
[19:46:38] <wikibugs>	 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), and 2 others: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3910896 (10awight) >>! In T181848#385272...
[19:48:19] <wikibugs>	 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), and 2 others: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3910901 (10awight)
[19:49:06] <wikibugs>	 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), and 2 others: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3910903 (10Adotchar) >>! In T181848#3910...
[19:49:28] <Adotchar>	 Hi
[19:49:47] <awight>	 Adotchar: hello, thanks for watching this task :)
[19:50:04] <Adotchar>	 I get emails when it’s changed
[19:50:11] <wikibugs>	 10Scoring-platform-team, 10Collaboration-Community-Engagement, 10MediaWiki-extensions-ORES, 10Patch-For-Review, 10User-notice-collaboration: Deploy ORES filters to Simple Wikipedia - https://phabricator.wikimedia.org/T182012#3910908 (10awight)
[19:50:15] <wikibugs>	 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), and 2 others: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3910905 (10awight) 05Open>03Resolved...
[19:50:21] <awight>	 Adotchar: Can you explain where you’ve been testing?
[19:50:36] <awight>	 i.e., are you using in a way that the enwiki thresholds are applied?
[19:50:49] <Adotchar>	 Halfak told me to put a few lines of text into a user page
[19:51:03] <awight>	 aha, on enwiki?
[19:51:06] <Adotchar>	 That evidently is English-Wikipedia’s ORES
[19:51:17] <Adotchar>	 No, one one of my simple wiki userpages
[19:51:20] <Zppix>	 halfak: T185148 good to close?
[19:51:21] <stashbot>	 T185148: Update docs, monitoring, etc. for new labs servers - https://phabricator.wikimedia.org/T185148
[19:51:48] <awight>	 Adotchar: Sorry, can you send me a link to where you read the scores?
[19:52:54] <Adotchar>	 Where I read the scores?
[19:53:03] <Adotchar>	 The ORES scores are available to me in the doffs
[19:53:06] <Adotchar>	 *diffs
[19:53:08] <awight>	 oho
[19:53:13] <awight>	 Using a gadget?
[19:53:17] <Adotchar>	 And it highlights edits in recent changes
[19:53:43] <Adotchar>	 Except it takes about 20 seconds to load in
[19:54:18] <awight>	 OK, thanks for describing.  That sounds like it was close enough to what we’re doing to deploy.
[19:54:22] <awight>	 *going
[19:54:50] <Adotchar>	 Okay
[19:54:56] <Adotchar>	 I’ve gtg now
[19:55:02] <wikibugs>	 10Scoring-platform-team (Current), 10Bad-Words-Detection-System, 10revscoring, 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), and 2 others: Experiment with using English Wikipedia models on Simple English - https://phabricator.wikimedia.org/T181848#3910917 (10awight)
[19:55:04] <wikibugs>	 10Scoring-platform-team, 10MediaWiki-extensions-ORES, 10ORES, 10Documentation: Elaborate documentation on how to deploy ORES to a new wiki - https://phabricator.wikimedia.org/T182054#3910916 (10awight)
[19:55:10] <awight>	 o/
[20:26:54] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Release-Engineering-Team: Beta ORES deployment fails with SSH host key mismatch - https://phabricator.wikimedia.org/T185254#3911004 (10awight)
[20:27:45] <wikibugs>	 (03PS3) 10Awight: Build venv into versioned source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071)
[20:32:23] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Release-Engineering-Team: Beta ORES deployment fails with SSH host key mismatch - https://phabricator.wikimedia.org/T185254#3911024 (10awight) 05Open>03Resolved a:03awight I sudo'd and removed the old host keys manually.
[20:36:35] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Release-Engineering-Team, 10Scap: scap support for multiple services - https://phabricator.wikimedia.org/T185255#3911040 (10awight)
[20:38:18] <wikibugs>	 (03PS4) 10Awight: Build venv into versioned source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071)
[20:39:06] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Release-Engineering-Team, 10Scap: scap support for multiple services - https://phabricator.wikimedia.org/T185255#3911067 (10awight) p:05Triage>03Low
[20:42:35] <wikibugs>	 (03PS5) 10Awight: Build venv into versioned source dir [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/392682 (https://phabricator.wikimedia.org/T181071)
[20:56:05] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Scap: scap support for multiple services - https://phabricator.wikimedia.org/T185255#3911116 (10greg)
[21:04:05] <Amir1>	 I'm leaving for the day
[21:04:08] <Amir1>	 see you tomrrow
[21:04:09] <Amir1>	 o/
[21:04:53] <awight>	 Amir1: oh I just realized that I forgot to remove the manual rules for your last PR
[21:04:56] <awight>	 Amir1: bye!
[21:05:10] <Amir1>	 I removed them in this PR :D
[21:05:20] <awight>	 nicely done
[22:28:27] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Scap, 10Release-Engineering-Team (Next): scap support for multiple services - https://phabricator.wikimedia.org/T185255#3911371 (10mmodell) +1 indeed that should be supported.
[23:02:25] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [extensions/ORES] - 10https://gerrit.wikimedia.org/r/405147 (owner: 10L10n-bot)