Linux, Virtualisation and some Performance monitoring

P.S.  This post is more of a noob's voyage towards better virtualisation on Linux than some professional guidance.
 
A few days ago I was facing an issue on my Linux machine where performance suddenly dropped to near unusability while the hard disk LED was on overdrive.  My first thought was that there may be some excessive swapping going on.  The problem was, though, how to identify what was causing this rather than what was happening.

Cheesy image for computers!

I could have guessed what the reason was since I had just turned on maybe the 10th VM on VMWare workstation.  Despite this fact it was not immediately obvious which VM might be swapping rapidly or why it was doing so (there shouldn't be much memory usage during startup).

As yet, I haven't totally found out what it was but messing with some config files did the trick up to a certain point.  First of all I limited VMWare to 16GB or RAM (out of 32) and configured it to swap as much as possible.  I was led to believe that VMWare's and the kernel's swapping mechanisms weren't on the same terms, which ended up with me bashing (excuse the pun) the whole system.  A few miraculous key presses took (Ctrl Alt F1) me to the terminal from where I could, at least, get a list of CPU heavy processes and kill them.  Unfortunately it was not just vmware-vmx processes but also kswapd0 - an integral part of the system which won't easily allow you to kill -9 it.  So basically this was the first indication of a memory issue.

After some googling I reconfigured swapping etc.  but I wasn't able to replicate the issue and quite frankly I really did not want to spend 15 everytime to recover my system.  So the process of finding a solution took days - not continuously trying to fix it of course.  the best solution I could come up with was buying a small 50GB SSD and using it all for swapping.  Apart from that I also set the vm.swappiness to a nice 100.  The memory configuration on VMWare was set to swap as much as possible too.  My idea was to allow everything to swap as much as they can since the disk was much faster now.  Apart from that, I'd have as little occupied memory as possible.

I thought I'd start seeing a lot of fast swapping this time, so in case I got into the same situation again, it would be much easier to recover.  In fact it did happen once again, but this time the system was under much more stress, so the extra swapping did help.  This time I had a little script prepared so the 10 second long keypresses would not waste much of my time when typing in all the arguments.  I used the following script to see what was hogging the CPU, network, disks - almost every possible bottleneck I could think of:

#!/bin/bash
dstat -cdnpmgs --top-bio --top-cpu --top-mem

Short and sweet, just calling dstat with canned arguments!  Calling jtop is a lot shorter than all those arguments, that's for sure.  Again, the result was swapping a swapping issue. 

dstat however showed me something I was not really expecting.  RAM usage wasn't really that bad, actually, just by looking at numbers it was great - less than 50% usage.  However there were some more numbers and at that point I was not sure if I was actually using ~40% or 97%.

Reading up on Linux memory management taught me another thing.  Linux is actually making use of much more RAM, however the bulk of it is caching.  This cache is cleared when more memory usage is required by processes.  Effectively I would see that there is less than 2-3% free RAM but that is not the correct way to read it.  So there is some silver lining to this issue - I got to learn quite a lot more on memory management on Linux.

Following this result I started looking for a virtualisation solution that did not try to re-implement what the kernel was built to do.  Not that I have anything in particular against VMWare or how it is implemented, but I was quite sure that the problem was originating from it.  After a bit more educated reading on virtualisation, and a bit more courage to move out my (then) GUI-based comfort zone (few weeks before the said case I was mostly a Windows user..), I came to the conclusion that the Linux-based systems were potentially much better.

The logo is cool though

Here I introduced myself to KVM and Xen.  Both appear to be more ingrained into the system and had potentially better memory management.  I read up on some general performance and history of both systems and KVM appeared to have the upper hand.  Being a more integral part of the Linux eco-system (and marginally faster https://major.io/2014/06/22/performance-benchmarks-kvm-vs-xen/) I opted to base my future VMs on KVM.  I'm happy to say that I've never looked back since then and the impressive performance I enjoy on KVM is (on my system) unparalleled.

I'll let the kernel manage my memory

There is no particular conclusion here, except that maybe you should be familiar with your options before making decisions.  I've got nothing against VMWare, as I said, I simply found something that works better for me.  Management tools are far better on the VMWare side, but I'm satisfied with what VM Manager offers in terms of management and monitoring.  Oh, you may also make use of the "script" I have.  It's convenient when you need to see some performance details while not keying in some 5 arguments.  I might write something on KVM next time, since it allows one to define many more options rather than a few clicks and done.

Hosting static websites on AWS S3

Hosting websites nowadays has become quite simple, easy and affordable.  Years ago you would try to find free hosts which would allow you to upload a few files and you're done.  GeoCities and freewebs were some of the most popular of these services.  As time went by, the data centre landscape has changed dramatically and the current situation is big companies offering extremely cheap hosting services.  The market is so huge that the classic "brochure site" has become almost free to host while still enjoying world class availability and scalability.

Static websites are the simplest form of site.  These are just a set of HTML pages linked together via links.  No complicated code, no weird configurations - just plain old files.  Depending on what you want, such a site may be ideal (maybe you do all your business transactions over a facebook page and use the site just as a brochure).

This of course has the advantage of being simple, easy, cheap and can be up and running very quickly, including development time.  It lacks, however, some of the major features you may want, such as a member's area, blogs and news, user-generated content etc.  But then again, you might not want these extra features.  In that case, here is a simple, short and sweet guide on how to host your site on Amazon Web Services.  There is no doubt that it is currently the leader in cloud services and it would be wise to use their services.

1. Get A Domain

The first thing you need is the dot-com.  You may use or favourite registrar or just randomly choose one from these that come to mind: noip.com, namecheap.com, godaddy.com.  If this is your first time you may want to read up on registration etc. but all you need is to find a domain that is available, buy it, and configure it as explained later.  Make sure you do not buy hosting with it, as some providers will try to bundle them together.  Well you can do whatever you want, but it's not necessary in this case.

2. Sign Up with AWS

Log on to aws.amazon.com and sign up for an account.  Choose your region, etc and keep going until you get to the main control panel.

3. Hold your horses!

The control panel may be overly complicated.  It isn't, though.  The number of services may be overwhelming, including the names, but we'll get through.  Only two services are required in our case.

Cloud providers typically offer more than just simple hosting.  Keep in mind that big enterprises are running their businesses here too, so this complexity is to be expected.  One of the core offerings of a cloud provider is storage.  Storage keeps everything in place - services need to save their logs, applications exist in storage, databases are persisted to storage...you get the pattern.  Again, due to the enterprisiness of this offering, the storage services have their own terminology.

Your usual "hard-disk" or "USB drive" (or floppy disk) is known as a bucket.  You have a bucket in the cloud in which you put your files.  Amazon offers storage in a service known as S3 - Simple Storage Service.  These bucket also tend to be dirt cheap.  A site a less than 10 pages and low to moderate traffic may cost you no more than €1 a month.

4. Creating the site

Now that you know about the basic concept, it is time to create the storage for your site.  In this example (and pretty much any other tutorial out there), we shall use the example.com domain.  Whenever you see this written down, replace it with the domain name you bought.  Do not prepend it with "www."; that is a subdomain, not the proper domain that you bought.

4.a. Sign in to https://console.aws.amazon.com/s3/;
4.b. Create a bucket named example.com;
4.c. Create another bucket www.example.com (with the www);
4.d. Upload your content to the first (example.com) bucket;

What we'll do is host the site on the example.com bucket and redirect any traffic coming in to www.example.com to it.

5. Prepare the site

Now you'll need to allow the public to access your buckets, otherwise they'll be forbidden from seeing your content (which, presumably, you want to be publicly accessible).  All you need is to attach the following bucket policy to your example.com bucket.  Again, make sure you replace example.com with your domain.

5.a. Set the policy
{
  "Version":"2012-10-17",
  "Statement":[{
"Sid":"AddPerm",
        "Effect":"Allow",
 "Principal": "*",
      "Action":["s3:GetObject"],
      "Resource":["arn:aws:s3:::example.com/*"
      ]
    }
  ]
}

5.b. Set up as static website by clicking on the bucket and select the Static Website Hosting section.  Choose the 'Enable' option;
5.c. Name the index page.  This is the "homepage" of your site.  Typically this is named "index.html" or similar;
5.d. Test the page by entering the Endpoint URL shown to you in your browser, just to make sure it is accessible;
5.e. Select the second bucket (www.example.com) and in the same section choose to redirect requests.  In the input field, enter your domain (without www.);

6. Wire it up

Another service that is required to properly route traffic to our site is Route 53.  As you've seen, you endpoint is appended with a much longer address belonging to amazon.  You wouldn't want to distribute that URL to your clients, after all you bought your own domain.

Route 53 is basically a DNS service - an internet directory for converting example.com to a number that the internet understands.  You do not need to do any of this works, except how to inform the registrar about your shining new website on AWS.  Here's how:

6.a. Open up https://console.aws.amazon.com/route53 and create a hosted zone for your domain (no www.) - Click Get Started Now under DNS Management, or just go to Hosted Zones and then Create Hosted Zone;
6.b. In the details section you'll see a Delegation Set - a list of addresses.  Write these down somewhere, we'll use them later on;
6.c. Click Create Record Set and enter you domain name.  Mark it as Alias and from the Alias target select your bucket.  Do this also for the www.example.com (point it to its own bucket).

7. Finishing

Now that everything is set up on AWS, all you need to do is inform the domain registrar (the site from where you bought your domain).  Remember the 4 addresses in the Delegation Set?  These will now be used to configure the DNS addresses for your domain.  What you need to do is log in to your domain registrar control panel and configure your domain.  Somewhere you should be able to change the DNS settings for it.  Not all providers have four fields - there may be more, there may be less.  Enter all four addresses in the delegation set in the domain configuration.  If there are less than four fields, that's it.  If there are more than four, leave the rest empty.

8.  Live!

Now that you're done, you may need to wait a few minutes until the DNS settings are updated.  This is not related to your site on AWS but on the nature of DNS - i.e. people's ability to enter example.com and be properly taken to your site.  These may take up to 48 hours, but in my case it was only a matter of minutes.

Hope you found this helpful!

HDMI over WiFi

No not really, no such thing exists.  Well you can stream HD media over WiFi but not in a plug and play way as a common HDMI cable.  There are various ways this can be accomplished but the idea is simple;  run a server from a device and play it on another, as long as there is a protocol, typically DLNA.

Recently I came across a new video format which looks quite interesting and powerful.  HEVC was released a few months ago (only drawn up sometime in 2013) and is now gaining traction.  I was surprised to find that an hour long HD video can be compressed down to 200 megabytes or so.  Being so new not many players can really decode it, as expected.  If you're on Linux, great news! It's not so hard to get it to play.  As always, VLC is your best friend so the solution I have/found is for VLC.  All that's needed is a new library from a PPA and you're good to go.

sudo apt-add-repository ppa:strukturag/libde265
sudo apt-get update
sudo apt-get install vlc-plugin-libde265

That takes no more than a few seconds; minutes if you're on a slow connection (which makes the HEVC format ideal in your case).

I wrote this little post about this format because I am quite interested in how the far we can compress HD videos, but also because it was a bit inconvenient for me to not be able to play it pretty much anywhere (BubbleUPnP and chromecast appear to handle it though).  It may spare you some time hunting for a way to watch that latest episode next time :)  Yet still, my original thought (and title) of this post were actually meant for something totally different, but hey, two birds one stone!

I need to learn to HDMI

Having organised cables is quite important if you intend to keep your sanity.  Not long ago I did rewire my homelab and after a few hours everything look perfectly in place.  Except for one thing; WiFi.  It was constantly disconnecting and apparently going offline and back again for no apparent reason.  Restarting, re-configuring and switching ports fixed nothing.

A few hours or days go by and I started noting red pixels showing up on my screen at particular points where the image was darker.  Then it hit me.  There must have been some interference between my WiFi router and the HDMI cable in its vicinity (hence the title ;)).  Looking around on the internet seemed to prove my theory, even though similar cases appeared to be quite uncommon.  I have not completely fixed it yet however messing around with the cable is a good workaround.  That's until the shielded HDMI cable arrives in my mailbox, which should hopefully fix the issue.