[00:12:32] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 27% free memory [00:18:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:26:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:49:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [00:53:02] bots & Labs users who love Python: PyCon 2013 is March in Silicon Valley: https://us.pycon.org/2013/ If you want to go and maybe lead a dev sprint? there's financial support available: https://meta.wikimedia.org/wiki/Participation:Support [00:56:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:07:42] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.46, 3.94, 4.86 [01:08:32] PROBLEM Total processes is now: WARNING on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS WARNING: 173 processes [01:13:33] RECOVERY Total processes is now: OK on bots-salebot i-00000457.pmtpa.wmflabs output: PROCS OK: 98 processes [01:19:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:25:42] PROBLEM Current Load is now: WARNING on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: WARNING - load average: 8.32, 7.14, 5.91 [01:27:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:30:53] PROBLEM Current Load is now: WARNING on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: WARNING - load average: 6.26, 5.99, 5.25 [01:50:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [01:57:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:05:32] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 150 processes [02:21:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:27:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:41:53] PROBLEM Free ram is now: WARNING on wikistream-1 i-0000016e.pmtpa.wmflabs output: Warning: 14% free memory [02:51:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [02:55:53] RECOVERY Current Load is now: OK on parsoid-roundtrip6-8core i-000004f8.pmtpa.wmflabs output: OK - load average: 2.65, 4.00, 4.84 [02:57:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:10:28] 12/05/2012 - 03:10:28 - Updating keys for mwang at /export/keys/mwang [03:15:32] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 17% free memory [03:21:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:21:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 11% free memory [03:26:52] RECOVERY Current Load is now: OK on ve-roundtrip2 i-0000040d.pmtpa.wmflabs output: OK - load average: 2.82, 3.92, 4.67 [03:28:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:30:33] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 36% free memory [03:30:43] RECOVERY Current Load is now: OK on parsoid-roundtrip3 i-000004d8.pmtpa.wmflabs output: OK - load average: 4.83, 4.28, 4.77 [03:51:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [03:58:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:02:43] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.01, 4.63, 4.97 [04:13:33] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 18% free memory [04:21:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:28:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:51:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [04:58:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:22:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:29:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:36:42] RECOVERY SSH is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [05:36:52] RECOVERY dpkg-check is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: All packages OK [05:37:12] RECOVERY Disk Space is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: DISK OK [05:37:52] RECOVERY Current Load is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK - load average: 6.81, 5.99, 3.32 [05:38:52] RECOVERY Free ram is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK: 671% free memory [05:39:22] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 105 processes [05:40:32] RECOVERY Current Users is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: USERS OK - 0 users currently logged in [05:52:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [05:59:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:07:23] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 161 processes [06:23:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:25:29] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=613118 edit summary: /* Requests */ moved some approved requests to archive [06:25:43] PROBLEM Current Load is now: WARNING on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: WARNING - load average: 4.95, 5.37, 5.11 [06:27:23] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 206 processes [06:28:33] PROBLEM Total processes is now: CRITICAL on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS CRITICAL: 204 processes [06:29:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:29:33] PROBLEM Total processes is now: WARNING on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS WARNING: 154 processes [06:30:42] RECOVERY Current Load is now: OK on parsoid-roundtrip7-8core i-000004f9.pmtpa.wmflabs output: OK - load average: 4.12, 4.61, 4.86 [06:31:32] PROBLEM Total processes is now: WARNING on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS WARNING: 151 processes [06:37:43] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=613121 edit summary: /* User:Chihonglee */ question re shellname [06:39:02] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=613123 edit summary: /* User:SHL */ archiving [06:45:33] PROBLEM dpkg-check is now: CRITICAL on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:46:38] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=613131 edit summary: /* User:Avocato */ [06:47:43] PROBLEM Disk Space is now: CRITICAL on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: DISK CRITICAL - free space: / 5 MB (0% inode=48%): [06:48:20] Change on 12mediawiki a page Developer access was modified, changed by Sharihareswara (WMF) link https://www.mediawiki.org/w/index.php?diff=613133 edit summary: /* User:Edinwiki */ [06:48:33] PROBLEM Total processes is now: WARNING on incubator-apache i-00000211.pmtpa.wmflabs output: PROCS WARNING: 199 processes [06:49:33] RECOVERY Total processes is now: OK on parsoid-spof i-000004d6.pmtpa.wmflabs output: PROCS OK: 149 processes [06:49:53] PROBLEM dpkg-check is now: CRITICAL on puppetdoc2 i-0000052b.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [06:50:22] RECOVERY dpkg-check is now: OK on mw1-21beta-lucid i-00000416.pmtpa.wmflabs output: All packages OK [06:53:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [06:54:52] RECOVERY dpkg-check is now: OK on puppetdoc2 i-0000052b.pmtpa.wmflabs output: All packages OK [06:56:33] RECOVERY Total processes is now: OK on vumi-metrics i-000004ba.pmtpa.wmflabs output: PROCS OK: 147 processes [06:57:42] PROBLEM Disk Space is now: CRITICAL on testing-arky i-0000033b.pmtpa.wmflabs output: DISK CRITICAL - free space: / 13 MB (1% inode=48%): [06:58:12] PROBLEM Disk Space is now: CRITICAL on conventionextension-trial i-000003bf.pmtpa.wmflabs output: DISK CRITICAL - free space: / 13 MB (1% inode=48%): [06:59:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:23:34] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:29:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:54:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [07:59:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:23:52] RECOVERY Current Load is now: OK on parsoid-roundtrip4-8core i-000004ed.pmtpa.wmflabs output: OK - load average: 4.27, 4.34, 4.90 [08:24:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:29:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [08:56:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [09:01:33] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:01:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [09:16:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [09:24:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:31:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [09:54:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:02:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:24:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:32:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [10:55:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:02:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:11:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [11:16:52] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [11:25:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:32:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [11:37:22] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 112 processes [11:41:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [11:55:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:00:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 164 processes [12:03:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:20:24] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 211 processes [12:26:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:33:13] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [12:57:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:04:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:27:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:34:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [13:50:23] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 117 processes [13:57:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:04:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:28:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:33:23] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 181 processes [14:34:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [14:38:23] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 201 processes [14:40:28] 12/05/2012 - 14:40:28 - Updating keys for mwang at /export/keys/mwang [14:59:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:04:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:10:27] 12/05/2012 - 15:10:27 - Updating keys for mwang at /export/keys/mwang [15:15:05] 12/05/2012 - 15:15:05 - Creating a project directory for testproject-mike [15:29:12] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:29:24] <^demon> hashar: I put all my gerrit jobs back into jenkins :) [15:29:26] <^demon> https://integration.mediawiki.org/ci/view/Java/ [15:29:36] ^demon: oh noo [15:29:46] ^demon: I am going to disable the Gerrit trigger plugin :-] [15:29:53] <^demon> I'm not using the trigger plugin. [15:29:57] ahh nice [15:30:15] <^demon> the plugins just depend on core, and core just runs once per day unless I do it manually. [15:33:54] <^demon> I need a pre-build step that will pull in a specific patch before building. I guess I could just execute a shell command to `cd $WORKSPACE; git pull origin {foo}`? [15:34:02] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [15:35:30] ^demon: hmm [15:35:45] ^demon: IIRC you can ask the git plugin to merge in a branch after it got the change from upstream [15:37:13] ^demon: yeah that is well hidden under Git plugin > Merge options [15:37:30] ^demon: you can tick a "Merge before build" checkbox that let you specify the branch to merge to [15:37:38] ^demon: DOC: These options allow you to perform a merge to a particular branch before building. For example, you could specify an integration branch to be built, and to merge to master. In this scenario, on every change of integration, Jenkins will perform a merge with the master branch, and try to perform a build if the merge is successful. It then may push the merge back to the remote repository if the Git Push post-build action is selected. [15:38:03] ok that is maybe the opposite of what you want to do though [15:38:33] <^demon> No, that's not what I want. What I'm wanting is "get master, then pull this un-merged patch" [15:38:42] <^demon> I can probably hack up a pre-build step. [15:39:16] Execute shell: git merge mypatch [15:39:18] ? ;) [15:44:20] does it matter much if root is compromised on an instance by accident? [15:45:10] liangent: yeah it is a serious issue [15:45:29] liangent: that person basically own the instance and could attack other instances from there [15:46:05] hashar: how can I attack others? [15:46:31] I meant other instances in a project [15:46:32] PROBLEM Disk Space is now: WARNING on labs-nfs1 i-0000005d.pmtpa.wmflabs output: DISK WARNING - free space: /export 907 MB (5% inode=50%): /home 907 MB (5% inode=50%): [15:46:40] and overall it could do whatever it want [15:48:07] ok it doesn't sound that terrible [15:48:28] I'm just running some commands under root when it's not required [15:48:34] ah [15:48:35] only for convinence [15:48:44] I never use root :-D [15:48:59] (or almost never) [15:50:05] ^demon: are the gerrit job to create tarballs ? [15:50:27] <^demon> No, gerrit doesn't make tars. [15:50:31] <^demon> Or need them. [15:51:07] so what the jobs are for ? :) [15:51:39] <^demon> We build wars and jars :) [15:52:17] ^demon: I think analytics is willing to set up something like that [15:52:38] <^demon> Why would analytics build the wars for gerrit? [15:52:40] ^demon: and they want a maven repository to publish the *ar [15:52:48] I meant, .wars for their own projects [15:52:51] <^demon> I'm not doing anything with maven repos. [15:52:55] <^demon> I hate maven enough as it is. [15:53:03] <^demon> I'm not deploying to manganese/formey from maven. [15:53:13] hehe [15:53:31] <^demon> Gerrit builds the wars, which is *exactly what I want it to do* [15:53:35] <^demon> Nothing more :) [15:54:02] you can copy the result somewhere under /srv/org/mediawiki/integration/nightly/ [15:54:18] then they will be available at http://integration.mediawiki.org/nightly/ [15:54:32] one day I will tweak the apache conf so the pages look a bit nicer [15:56:08] ^demon: also andrew started building puppet documentation at http://doc.wikimedia.org/ :) [15:56:26] might end up migrating doxygen out of formey to gallium :) [15:56:41] <^demon> Good. [15:56:46] <^demon> I want that off formey, badly. [15:56:54] we are close to be able to do that [15:57:01] I need to write down the jenkins job in Jenkins Job builder [15:57:10] then run it once per day [15:57:18] or maybe on any merge, I don't know [15:57:31] I also want the doc for 1.19 / 1.20 to be generated there [15:59:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:04:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:08:22] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 121 processes [16:26:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 173 processes [16:29:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:32:53] RECOVERY Free ram is now: OK on aggregator1 i-0000010c.pmtpa.wmflabs output: OK: 725% free memory [16:34:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [16:36:23] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 213 processes [16:51:54] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 10% free memory [17:00:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:04:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:24:17] benestar: what's the status? [17:24:35] no change to yesterday :/ [17:26:25] i hope that situation will change somewhen in the future [17:30:23] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:34:03] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [17:41:50] * jeremyb waves+welcomes mike_wang [17:57:55] Change on 12mediawiki a page Developer access was modified, changed by Alexander.dolidze link https://www.mediawiki.org/w/index.php?diff=613324 edit summary: /* Please create git user to create access: https://gerrit.wikimedia.org/r/translatewiki */ new section [17:59:11] Change on 12mediawiki a page Developer access was modified, changed by Alexander.dolidze link https://www.mediawiki.org/w/index.php?diff=613325 edit summary: /* Please create git user to create access: https://gerrit.wikimedia.org/r/translatewiki */ [17:59:51] Change on 12mediawiki a page Developer access was modified, changed by Alexander.dolidze link https://www.mediawiki.org/w/index.php?diff=613326 edit summary: /* User:Alexander.dolidze */ [18:01:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:01:52] PROBLEM Free ram is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Warning: 15% free memory [18:04:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:07:22] Change on 12mediawiki a page Developer access was modified, changed by Alexander.dolidze link https://www.mediawiki.org/w/index.php?diff=613331 edit summary: /* User:Alexander.dolidze */ [18:16:52] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Critical: 3% free memory [18:21:53] RECOVERY Free ram is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK: 540% free memory [18:24:33] PROBLEM Free ram is now: WARNING on ipv6test1 i-00000282.pmtpa.wmflabs output: Warning: 19% free memory [18:25:52] PROBLEM Current Load is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: WARNING - load average: 1.72, 14.65, 9.75 [18:26:22] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 129 processes [18:31:52] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:34:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [18:39:22] PROBLEM Total processes is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS WARNING: 179 processes [18:39:42] PROBLEM Free ram is now: CRITICAL on ipv6test1 i-00000282.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [18:40:53] RECOVERY Current Load is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK - load average: 2.52, 1.56, 4.03 [18:44:33] RECOVERY Free ram is now: OK on ipv6test1 i-00000282.pmtpa.wmflabs output: OK: 35% free memory [18:49:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS CRITICAL: 219 processes [19:01:53] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:04:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:13:12] PROBLEM dpkg-check is now: CRITICAL on integration-jenkins2 i-000004f6.pmtpa.wmflabs output: DPKG CRITICAL dpkg reports broken packages [19:32:42] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:36:33] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [19:46:18] Change on 12mediawiki a page Developer access was modified, changed by Felizardow link https://www.mediawiki.org/w/index.php?diff=613365 edit summary: [20:02:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:06:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:09:53] PROBLEM Free ram is now: WARNING on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Warning: 18% free memory [20:14:08] Change on 12mediawiki a page Developer access was modified, changed by Felizardow link https://www.mediawiki.org/w/index.php?diff=613386 edit summary: [20:24:52] PROBLEM Free ram is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Critical: 5% free memory [20:25:27] 12/05/2012 - 20:25:26 - Updating keys for mwang at /export/keys/mwang [20:26:53] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [20:32:43] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:34:23] PROBLEM Total processes is now: UNKNOWN on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: NRPE: Call to fork() failed [20:34:33] PROBLEM Current Users is now: UNKNOWN on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: NRPE: Call to fork() failed [20:34:53] PROBLEM Free ram is now: UNKNOWN on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: NRPE: Call to fork() failed [20:34:53] PROBLEM dpkg-check is now: UNKNOWN on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: NRPE: Call to fork() failed [20:36:13] PROBLEM Disk Space is now: UNKNOWN on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: NRPE: Call to fork() failed [20:36:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:36:53] PROBLEM Free ram is now: WARNING on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Warning: 6% free memory [20:37:07] Change on 12mediawiki a page Developer access was modified, changed by Felizardow link https://www.mediawiki.org/w/index.php?diff=613416 edit summary: /* User:Felizardow */ [20:38:53] PROBLEM Current Load is now: UNKNOWN on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: NRPE: Call to fork() failed [20:42:26] So, wikidata have a box that is like completely out of memory [20:42:41] Can someone hard reboot it somehow? Labsconsole fails [20:43:52] PROBLEM Current Load is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:44:22] PROBLEM Total processes is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:44:32] PROBLEM Current Users is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Error - Could not complete SSL handshake. [20:44:42] PROBLEM SSH is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: Server answer: [20:45:02] PROBLEM dpkg-check is now: CRITICAL on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: CHECK_NRPE: Socket timeout after 10 seconds. [20:45:23] PROBLEM host: i-0000052a.pmtpa.wmflabs is DOWN address: i-0000052a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [20:48:53] RECOVERY Current Load is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: OK - load average: 1.35, 0.33, 0.11 [20:48:53] Reedy: the connection to wikidata-dev-9 just broke [20:49:03] RECOVERY host: i-0000052a.pmtpa.wmflabs is UP address: i-0000052a.pmtpa.wmflabs PING OK - Packet loss = 0%, RTA = 3.55 ms [20:49:07] did any of you reboot the box? [20:49:23] RECOVERY Total processes is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: PROCS OK: 106 processes [20:49:31] looks much better now [20:49:33] RECOVERY Current Users is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: USERS OK - 1 users currently logged in [20:49:43] RECOVERY SSH is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [20:49:53] RECOVERY dpkg-check is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: All packages OK [20:51:12] RECOVERY Disk Space is now: OK on wikidata-dev-9 i-0000052a.pmtpa.wmflabs output: DISK OK [20:56:52] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 5% free memory [21:03:32] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:06:43] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:15:27] 12/05/2012 - 21:15:27 - Updating keys for mwang at /export/keys/mwang [21:33:33] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [21:37:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:04:02] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:07:22] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:27:22] PROBLEM Total processes is now: WARNING on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS WARNING: 152 processes [22:34:13] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:37:23] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [22:57:22] RECOVERY Total processes is now: OK on nova-precise1 i-00000236.pmtpa.wmflabs output: PROCS OK: 150 processes [23:02:06] PROBLEM Free ram is now: CRITICAL on dumps-bot1 i-000003ed.pmtpa.wmflabs output: Critical: 4% free memory [23:04:47] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:07:42] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:35:22] PROBLEM host: i-000004de.pmtpa.wmflabs is DOWN address: i-000004de.pmtpa.wmflabs PING CRITICAL - Packet loss = 100% [23:38:12] PROBLEM host: i-0000051a.pmtpa.wmflabs is DOWN address: i-0000051a.pmtpa.wmflabs PING CRITICAL - Packet loss = 100%