Categories
Uncategorized

new minidlna server broke my network

We added a new device which can expose a connected USB drive via DLNA, internally it uses minidlna which uses SSDP for service discovery. For some strange reason that rendered my *existing* minidlna (hosted on a raspberry pi) invisible. When researching the problem, it looks like neighbor discovery (which didn’t happen before as there were no other devices) uses a multicast 239.0.0.0/8 address which my rpi was blocking due to reasons (only allows traffic via the local network and a vpn gateway). My theory is that the new minidlna device took over as “primary” and then couldn’t find other peers and so the old server wasn’t visible anymore. The solution was to allow the specific multicast address used by SSDP.

#!/bin/bash
iptables -F
#Tunnel interface
iptables -A INPUT -i tun+ -j ACCEPT
iptables -A OUTPUT -o tun+ -j ACCEPT
#Localhost and local networks
iptables -A INPUT -s 127.0.0.0/16 -j ACCEPT
iptables -A OUTPUT -d 127.0.0.0/16 -j ACCEPT
iptables -A INPUT -s 192.168.0.0/16 -j ACCEPT
iptables -A OUTPUT -d 192.168.0.0/16 -j ACCEPT
#multicast for minidlna/SSSP
iptables -I OUTPUT -d 239.255.255.250 -j ACCEPT
iptables -I INPUT -d 239.255.255.250 -j ACCEPT
#Allow VPN establishment, this is the port in the config's #remote
iptables -A OUTPUT -p udp --dport 1198 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p udp --sport 1198 -m state --state ESTABLISHED,RELATED -j ACCEPT
#Drop everything else
iptables -A INPUT -j DROP
iptables -A OUTPUT -j DROP

Categories
English Geeky

Converting only one stream in a mkv file.

These mkv files have h.265 hevc video which my media player can’t read, so I’d like to convert only the video stream to h.264, while leaving all other streams (2 audio tracks in aac, 2 subtitle tracks) intact.

ffmpeg -i some-x265-video.mkv -map 0 -c:v libx264 -c:a copy /tmp/x264-version.mkv

Categories
English Geeky

Cross-timezone date calculations using the “date”command

Working remotely for a timezone-distributed company poses an interesting challenge: that of having to figure out dates and times for people in different timezones. This involves not only the relatively trivial “what time is it now in A_FARAWAY_PLACE”, but “what time, in FARAWAY_PLACE_X, will it be in FARAWAY_PLACE_Z” and other fun things.

There are a handful of websites that have handy tools to do these conversions for you; but a problem I’ve found is that the web is going to the crapper, and these sites often have confusing UIs concocted by some javascript-crazed, CSS-infected webmonkey; and often they are completely swamped and rendered unusable by a rising tide of ads and other aggressive content (oh and some won’t let you do anything until you agree to them storing information in cookies in your browser – which they then bafflingly don’t use to store the *PREFERENCE* you have selected , so like a forgetful vampire, they ask you every single time if you want to accept their silly cookies).

I’ve known how to use the “date” command to show the date on a different place/timezone, which is already a huge timesaver:

$ TZ="Taiwan/Taipei" date
Fri Apr 12 19:25:31 Taiwan 2019

but – today I was trying to answer “what time in TZ=”America/Chicago” is 1 PM, on Tuesday, in “UK/London“. This is interesting because it’s conversion between two timezones which are not the one I’m in, of a date/time in the future. So I was checking date’s man page for “how to convert a specific point in time”, when I realized date can do this for you! Right in the man page there’s this example:

Show the local time for 9AM next Friday on the west coast of the US

          $ date --date='TZ="America/Los_Angeles" 09:00 next Fri'

so then I combined that with the earlier one to come up with:

$ TZ="America/Chicago" date --date='TZ="UK/London" 1:00 PM next Tue'
Tue Apr 16 08:00:00 CDT 2019

This combines

  • TZ argument to calculate dates for a specific timezone, not the current one
  • –date parameter to “display time described by STRING, not ‘now'”
  • Descriptive time specifications (1:00 PM next Tuesday – this is a pseudo-human-readable format which is not entirely intuitive – info date has the specifics)
  • TZ support *inside* the descriptive specification

And a list of known timezones can be obtained with timedatectl list-timezones.

Categories
English Geeky

Remote display of a KVM virtual machine

In this case I’m hosting the VM on a fast server and trying to access the display on another system (a laptop).

One way to do it is by simply SSHing with X forwarding and running KVM like so:

qemu-system-x86_64 -boot d -cdrom ubuntu-18.04.2-live-server-amd64.iso -m 8192 -enable-kvm

This by default uses a terminal window, but it’s quite slow.

Another option is to start the KVM machine in nographic mode and enable a VNC server:

qemu-system-x86_64 -nographic -vnc :5 -boot d -cdrom ubuntu-14.04.6-desktop-amd64.iso -m 8192 -enable-kvm

then on the desktop system use a vnc client to connect to the magic port:

xtightvncviewer thehost.local:5905
Categories
English Geeky Uncategorized

KVM bridged to the LAN with DHCP

The goal here is to instantiate VMs with a br0 interface grabbing an IP from the LAN DHCP, so in turn the VM can instantiate LXD containers whose IP is also exposed to the LAN. That way everything is visible on the same network segment and this makes some experimentation easier.

Host configuration

Some info taken from this URL.

The metal host is running Ubuntu 18.04, which uses netplan. Here’s the netplan.yaml file:

network:
    ethernets:
        enp7s0:
            addresses: []
            dhcp4: no
            dhcp6: no
            optional: true
    bridges:
        br0:
            dhcp4: true
            dhcp6: no
            interfaces:
                - enp7s0
            parameters:
                stp: false
                forward-delay: 0
    version: 2

With this, on boot the system grabs an address from the network’s DHCP service (from my home router) and puts it on the br0 interface (which bridges enp7s0, a Gigabit Ethernet port).

The system also has avahi-daemon installed so I can ssh the-server.local easily.

VM configuration

Next, the VM which I created using uvt-kvm:

# Get a Xenial cloud image
uvt-simplestreams-libvirt --verbose sync release=xenial arch=amd64
# Create/launch a VM
PARAMS='--memory 8192 --disk 32 --cpu 4'
uvt-kvm create the-vm  $PARAMS --bridge br0 --packages avahi-daemon,bridge-utils,haveged --run-script-once setup_network.sh

The setup_network.sh script takes care of setting up the network 🙂 This can more cleanly be done with cloud-init but I’m lazy and wanted something fast.

The script deletes the cloudconfig-created .cfg file, tells cloud-init to NOT reconfigure the network, and drops the config file I actually need in place.

#!/bin/bash

echo "Acquire::http::Proxy \"http://192.168.1.187:3128\"; " >/etc/apt/apt.conf.d/80proxy

# Drop the cloudinit-configured interface
ifdown ens3

# Reconfigure the network...
cat </etc/network/interfaces.d/1-bridge.cfg
auto lo br0

iface lo inet loopback

iface ens3 inet manual

iface br0 inet dhcp
    bridge_ports ens3
    bridge_stp off       # disable Spanning Tree Protocol
    bridge_waitport 0    # no delay before a port becomes available
    bridge_fd 0          # no forwarding delay
EOF

echo "network: {config: disabled}" > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
rm /etc/network/interfaces.d/50-cloud-init.cfg

# Then bring up the new nice bridge
ifup br0

apt-get remove -y snapd && apt-get -y autoremove

 

The network config in /etc/network/interfaces.d/1-bridge.cfg should look like:

auto lo br0

iface lo inet loopback

iface ens3 inet manual

iface br0 inet dhcp
    bridge_ports ens3
    bridge_stp off       # disable Spanning Tree Protocol
    bridge_waitport 0    # no delay before a port becomes available
    bridge_fd 0          # no forwarding delay

LXD configuration

Finally,  install lxd. When asked to configure the lxd bridge, respond “no”, and on the next question you’ll be asked whether to supply an existing bridge. Respond “yes” and specify “br0”.

Now, when an lxd container is instantiated, it’ll by default appear on the same network (the home network!) as the VM and the main host, getting its DHCP from the home router.

When things break

Suddenly the bridge interface stopped working. I checked this to help diagnose it. But that wasn’t it. Turns out, I’d installed Docker on the main host and Docker messes with the firewall configuration by setting iptables -P FORWARD DROP. I just set it back to ACCEPT to get it working.

Categories
English Geeky

Bisecting Python unit test errors to find test interdependencies

 

Many of our test runs use parallelization to run faster. Sometimes we see test
failures which we can’t reproduce locally, because locally we usually run
sequentially; and even then, the test ordering seems to be somewhat
unpredictable so it’s hard to reproduce the exact test ordering seen in our
test runner.

Most of the time these failures are due to unidentified test interdependencies:
either test A causes test B to pass (where running test B in isolation would
fail), or test A causes B to fail (where running B in isolation would pass). And we have seen more complex scenarios where C passes, A-B-C passes, but A-C fails (because A sets C up for failure, while B would set C up for success). We added some diagnostic output to our test runner so it would show exactly the list of tests each process runs. This way we can copy the list and run it locally, which usually reproduces the failure.

But we needed a tool to then determine exactly which of the tests preceding the failing one was setting up the failure conditions. So I wrote this simple bisecter script, which expects a list of test names, which must contain the faily test “A”, and of course, the name of the faily test “A”. It looks for “A” in the list and will use bisection to determine which of the tests preceding “A” is causing the failure.

#!/usr/bin/python3
"""
Find which test in the test list is causing the failure of a known-failing
test. That is – Given a test list which dictates a specific test order,
under which a test X (which passes when run in isolation) is failing, find
out which of the tests that, on the list, run before X, are causing it to
fail.
Many of our test runs use parallelization to run faster. Sometimes we see test
failures which we can't reproduce locally, because locally we usually run
sequentially; and even then, the test ordering seems to be somewhat
unpredictable so it's hard to reproduce the exact test ordering seen in our
test runner.
Most of the time these failures are due to unidentified test interdependencies:
either test A causes test B to pass (where running test B in isolation would
fail), or test A causes B to fail (where running B in isolation would pass).
And we have seen more complex scenarios where C passes, A-B-C passes, but A-C
fails (because A sets C up for failure, while B would set C up for success).
We added some diagnostic output to our test runner so it would show exactly the
list of tests each process runs. This way we can copy the list and run it
locally, which usually reproduces the failure.
But we needed a tool to then determine exactly which of the tests preceding the
failing one was setting up the failure conditions. So I wrote this simple
bisecter script, which expects a list of test names, which must contain the
faily test "A", and of course, the name of the faily test "A". It looks for "A"
in the list and will use bisection to determine which of the tests preceding
"A" is causing the failure.
Note it's not very tunable, it will run "make test" with
ARGS='–failfast $LIST_OF_TESTS'
And interpret any non-zero exit code as "a test failed".
"""
import argparse
import math
import subprocess
import sys
def bisect_run(f_list, f_test):
# Always called with a f_list that causes f_test to fail.
if len(f_list) == 1:
return("The test that causes the failure is {}".format(f_list[0]))
if len(f_list) == 0:
return("No test causes the failure? what?")
first_half = f_list[:len(f_list)/2]
second_half = f_list[len(f_list)/2:]
print("{} elements in the list, about {} iterations left".format(
len(f_list), int(math.log(len(f_list), 2))))
try:
list_of_tests = first_half[:]
list_of_tests.append(f_test)
test_plan = " ".join(list_of_tests)
subprocess.check_output(
"make test ARGS='–failfast {}'".format(test_plan),
shell=True, stderr=subprocess.PIPE)
except:
print("Test causing failure is in first half of given list")
return bisect_run(first_half, f_test)
else:
print("Test causing failure is in second half of given list")
return bisect_run(second_half, f_test)
def main():
parser = argparse.ArgumentParser(description="""
Find which test in the test list is causing the failure of a known-failing
test. That is – Given a test list which dictates a specific test order,
under which a test X (which passes when run in isolation) is failing, find
out which of the tests that, on the list, run before X, are causing it to
fail.
""")
parser.add_argument("test_list", help="File containing a list of "
"test names, one per line.")
parser.add_argument("failing_test", help="Name of the test that fails. "
"It must exist in the test_list file.")
args = parser.parse_args()
with open(args.test_list, "r") as test_list_file:
test_list = [s.strip() for s in test_list_file.readlines()]
# We don't need to bother with tests before failing_test
f_index = test_list.index(args.failing_test)
test_list = test_list[:f_index1]
print(bisect_run(test_list, args.failing_test))
sys.exit(main())

view raw
bisecter.py
hosted with ❤ by GitHub

As an example, I used it to find a test failure in Ubuntu SSO:

python bisecter.py  test-orders/loadbad1.txt webui.tests.test_decorators.SSOLoginRequiredTestCase.test_account_must_require_two_factor
273 elements in the list, about 8 iterations left
Test causing failure is in second half of given list
137 elements in the list, about 7 iterations left
Test causing failure is in second half of given list
69 elements in the list, about 6 iterations left
Test causing failure is in first half of given list
34 elements in the list, about 5 iterations left
Test causing failure is in second half of given list
17 elements in the list, about 4 iterations left
Test causing failure is in second half of given list
9 elements in the list, about 3 iterations left
Test causing failure is in second half of given list
5 elements in the list, about 2 iterations left
Test causing failure is in second half of given list
3 elements in the list, about 1 iterations left
Test causing failure is in second half of given list
2 elements in the list, about 1 iterations left
Test causing failure is in first half of given list
The test that causes the failure is webui.tests.test_views_account.AccountTemplateTestCase.test_backup_device_warning

 

 

Categories
English House buying

Buying a house in Montreal – the credit report

A few of the resources I read mentioned it was a good idea to get a credit report prior to visiting the mortgage lender/broker, so any inaccuracies can be corrected in advance. They indicate the credit report can be obtained free of charge through Equifax or TransUnion.

I went to Equifax (note this was before their 2017 breach – I obviously now recommend you go to TransUnion instead). And their home page is a bit scary offering products protecting you from identity theft (I guess that didn’t help or protect those affected in 2017). But I was only interested in the credit report and score (the score is not necessary but it’s a nice single-number summary of your credit and I though it worth it). They have an option to purchase on-line for $23 so I went with that.

But remember the report can be obtained for free? understandably, this is not terribly visible in their front page, but there it is: “You can receive a free credit file disclosure from Equifax Canada Co. via Canada Post“. That link will take you to a form you can fill out and mail or fax (Fax, really?). So for the cost of a stamp and a bit of waiting you can also have your free credit report.

I was impatient and paid the $23, only to get a scary error when trying to get the report, which necessitated calling Equifax, in the course of the call they tried to upsell me on their credit report monitoring service (it’s cheaper, they said, only $16/month instead of the $23 you’re paying – conveniently not mentioning than the $23 is a one-shot charge). Which product you choose is up to you, just remember to stand your ground if you’re not interested in the more expensive one.

The credit report indicated a reasonably high score and no unusual activity, and should look good to any prospective lenders, so this phase is complete and we’re good to go!

Categories
English House buying

Buying a house in Montreal – the stress test

One of the changes to mortgage rules for 2016 in Canada is the creation of the “stress test“, meant to cool down hot real estate markets and keep people from buying houses that stretch their financial capabilities.

If you’re going for a high-ratio (less than 20% down payment) mortgage, lenders are required by law to check your payment capacity as if your loan interest rate were as high as the standard five-year rate (which currently is 4.94%), even if your eventual mortgage will actually be at a much lower (currently around 2-2.5%) rate.

The FCAC calculator makes it very easy to check what your maximum loan will be, once this rule is taken into account. Just enter your information and your expected interest rate to calculate your real maximum mortgage. Next, change the interest rate to 4.95% (I went super safe and put in 5%). It will tell you you won’t qualify, but you can now play with the maximum property value until it shows you you’re likely to be approved.

In my case, it resulted in a reduction of 18% in the maximum price I could afford, which is not terrible because all my previous calculations were taking this into account. Some people may be surprised, and discouraged out of the house hunting process by this, but if you know about this rule and factor it in your calculations prior to starting the process, you’ll know what to expect and how to compensate (get more money, save up for a larger down payment, lower your house price range).

 

Categories
English House buying

Buying a house in Montreal – where to start?

So we decided to buy a house, what will the journey look like?

There are plenty of easily-googlable resources on the house buying process in Canada and in Québec more specifically (here’s the two most detailed I’ve seen: FCAC and CMHC), so I won’t try to repeat that information, but I’ll document the specifics of our process which will invariably result in a brief outline of the steps to follow.

Roughly what we’ll try to do:

  1. Get a relatively good family income so we can qualify for a reasonable mortgage loan.
  2. Build up a credit history.
  3. Save up for a down payment.

We’ve worked on those first three steps since we moved to Canada: I’ve been fortunate enough to have a stable and well-paid job, which has allowed us to use consumer credit responsibly, so should have a pretty good rating. It also allowed us to save for a down payment. So at this point we should be ready for the next parts of the process:

  1. See a lender to get financially checked and pre-approved for a loan. You can go for a well-known financial institution, perhaps your bank, or you can go to a mortgage broker, which is what I’m planning on doing.
  2. Once you know your price range, you can start looking at houses in your desired areas.

BUT before you can start with this, you should know roughly how much you can afford, be realistic with your inputs and use one of the available online calculators. I like this one which will tell you how much you should be able to afford, and this one which calculates your estimated payments. And this one is very simple but also very detailed as to the criteria used to estimate affordability. It makes sense to use this so you’re not disappointed when the broker tells you you can only afford a tiny shack in the boondocks :).

You should also have a pretty good idea of whether you like your target neighbourhood. Montreal is a geographically large city and neighbourhoods can differ, so it makes sense to check the ones you like and make a short list. If you don’t care where you buy, there’s something for almost any price range, but I don’t think that’s very common.

A possible problem with the neighbourhood you like is whether you can afford it. If you can’t just yet, there are two options: choose a different one or get more money (higher salary, larger down payment).

Once I identified our target neighbourhoods, I started scouring centris.ca frequently, looking for houses in (and out of) our price range, checking their pictures and prices, nearby amenities, and comparing several possible neighbourhoods. We ended up discarding one of those, even though it was cheaper and had more inventory, because we decided we didn’t really like it that much. So we’re focusing on one of the other candidates, and also looking at adjacent neighbourhoods, which can be cheaper while still being closer to the amenities we want.

OK, so knowing how much we can afford (per the calculators) having located (and lived in) a neighborhood we like and knowing the approximate price range for homes here, and knowing it is within our affordability, I’m ready to hit the mortgage broker.

Resources:

http://www.fcac-acfc.gc.ca/Eng/resources/publications/mortgages/Pages/home-accueil.aspx

Categories
English Trabajo

Juju2 unit/service name autocompletion.

If juju1 and juju2 are installed on the same system, juju1’s bash auto completion breaks because it expects services where in juju2 they’re called applications.

Maybe juju2 has correct bash completion, but in the system I’m working on, only juju1 autocompletion was there, so I had to hack the autocomplete functions. Just added these at the end of .bashrc to override the ones in the juju1 package. Notice they work for both juju1 and juju2 by using dict.get() to not die if a particular key isn’t found.

 

# Print (return) all units, each optionally postfixed by $2 (eg. 'myservice/0:')
_juju_units_from_file() {
python -c '
trail="'${2}'"
import json, sys; j=json.load(sys.stdin)
all_units=[]
for k,v in j.get("applications", j.get("services",{})).items():
    if v.get("units"):
        all_units.extend(v.get("units",{}).keys())
print "\n".join([unit + trail for unit in all_units])
' < ${1?}
}

# Print (return) all services
_juju_services_from_file() {
python -c '
import json, sys; j=json.load(sys.stdin)
print "\n".join(j.get("applications", j.get("services",{}).keys());' < ${1?}
}

 

Categories
English Geeky

Take me to your leader – Using Juju leadership for cron tasks in a multiunit service

I’m working on adding some periodic maintenance tasks to a service deployed using Juju. It’s a standard 3-tier web application with a number of Django application server units for load balancing and distribution.

Clearly the maintenance tasks’ most natural place to run is in one of these units, since they have all of the application’s software installed and doing the maintenance is as simple as running a “management command” with the proper environment set up.

A nice property we have by using Juju is that these application server units are just clones of each other, this allows scaling up/down very easily because the units are treated the same. However, the periodic maintenance stuff introduces an interesting problem, because we want only one of the units to run the maintenance tasks (no need for them to run several times). The maintenance scripts can conceivably be run in all units, even simultaneously (they do proper locking to avoid stepping on each other). And this would perhaps be OK if we only had 2 service units, but what if, as is the case, we have many more? there is still a single database and hitting it 5-10 times with what is essentially a redundant process sounded like an unacceptable tradeoff for the simplicity of the “just run them on each unit” approach.

We could also implement some sort of duplicate collapsing, perhaps by using something like rabbitmq and celery/celery beat to schedule periodic tasks. I refused to consider this since it seemed like swatting flies with a cannon, given that the first solution coming to mind is a one-line cron job. Why reinvent the wheel?

The feature that ended up solving the problem, thanks to the fine folks in Freenet’s #juju channel, is leadership, a feature which debuted in recent versions of Juju. Essentially, each service has one unit designated as the “leader” and it can be targeted with specific commands, queried by other units (‘ask this to my service’s leader’) and more importantly, unambiguously identified: a unit can determine whether it is the leader, and Juju events are fired when leadership changes, so units can act accordingly. Note that leadership is fluid and can change, so the charm needs to account for these changes. For example, if the existing leader is destroyed or has a charm hook error, it will be “deposed” and a new leader is elected from among the surviving units. Luckily all the details of this are handled by Juju itself, and charms/units need only hook on the leadership events and act accordingly.

So it’s then as easy as having the cron jobs run only on the leader unit, and not on the followers.

The simplistic way of using leadership to ensure only the leader unit performs an action was something like this in the crontab:

* * * * * root if [ $(juju-run {{ unit_name }} is-leader) = 'True' ]; then run-maintenance.sh; fi 

This uses juju-run with the unit’s name (which is hardcoded in the crontab – this is a detail of how juju run is used which I don’t love, but it works) to run the is-leader command in the unit. This will print out “True” if the executing unit is the leader, and False otherwise. So this will condition execution on the current unit being the leader.

Discussing this with my knowledgeable colleagues, a problem was pointed out: juju-run is blocking and could potentially stall if other Juju tasks are being run. This is possibly not a big deal but also not ideal, because we know leadership information changes infrequently and we also have specific events that are fired when it does change.

So instead, they suggested updating the crontab file when leadership changes, and hardcoding leadership status in the file. This way units can decide whether to actually run the command based on locally-available information which removes the lock on Juju.

The solution looks like this, when implemented using Ansible integration in the charm. I just added two tasks: One registers a variable holding is-leader output when either the config or leadership changes:

    - name: register leadership data
      tags:
        - config-changed
        - leader-elected
        - leader-settings-changed
      command: is-leader
      register: is_leader

The second one fires on the same events and just uses the registered variable to write the crontabs appropriately. Note that Ansible’s “cron” plugin takes care of ensuring “crupdate” behavior for these crontab entries. Just be mindful if you change the “name” because Ansible uses that as the key to decide whether to update or create anew:

    
- name: create maintenance crontabs
      tags:
        - config-changed
        - leader-elected
        - leader-settings-changed
      cron:
        name: "roadmr maintenance - {{item.name}}"
        special_time: "daily"
        job: "IS_LEADER='{{ is_leader.stdout }}'; if [ $IS_LEADER = 'True' ]; then {{ item.command }}; fi"
        cron_file: roadmr-maintenance
        user: "{{ user }}"
      with_items:
        - name: Delete all foos
          command: "delete_foos"
        - name: Update all bars
          command: "update_bars"

A created crontab file (in /etc/cron.d/roadmr-maintenance) looks like this:

# Ansible: roadmr maintenance - Delete all foos
@daily roadmr IS_LEADER='True'; if [ $IS_LEADER = 'True' ]; then delete_foos; fi

A few notes about this. The IS_LEADER variable looks redundant. We could have put it directly in the comparison or simply wrote the crontab file only in the leader unit, removing it on the other ones. We specifically wanted the crontab to exist in all units and just be conditional on leadership. IS_LEADER makes it super obvious, right there in the crontab, whether the command will run. While redundant, we felt it added clarity.

Save for the actual value of IS_LEADER, the crontab is present and identical in all units. This helps people who log directly into the unit to understand what may be going on in case of trouble. Traditionally people log into the first unit; but what if that happens to not be the leader? If we write the crontab only on the leader and remove from other units, it will not be obvious that there’s a task running somewhere.

Charm Ansible integration magically runs tasks by tags identifying the hook events they should fire on. So by just adding the three tags, these events will fire in the specified order on config-changed, leader-elected and leader-settings-changed events.

The two leader hooks are needed because leader-elected is only fired on the actual leader unit; all the others get leader-settings-changed instead.

Last but not least, on’t forget to also declare the new hooks in your hooks.py file, in the hooks declaration which now looks like this (see last two lines added):

hooks = charmhelpers.contrib.ansible.AnsibleHooks(
    playbook_path='playbook.yaml',
    default_hooks=[
        'config-changed',
        'upgrade-charm',
        'memcached-relation-changed',
        'wsgi-file-relation-changed',
        'website-relation-changed',
        'leader-elected',
        'leader-settings-changed',
    ])

Finally, I’d be remiss not to mention an existing bug in leadership event firing. Because of that, until leadership event functionality is fixed and 100% reliable, I wouldn’t use this technique for tasks which absolutely, positively need to be run without fail or the world will end. Here, I’m just using them for maintenance and it’s not a big deal if runs are missed for a few days. That said, if you need a 100% guarantee that your tasks will run, you’ll definitely want to implement something more robust and failproof than a simple crontab.

Categories
Uncategorized

How to configure e-mail alerts with Munin

I had a hell of a time configuring Munin to send out e-mail alerts if values surpass specific thresholds. Many of the articles I found focused just on setting up the email command (which was the easy part), while few told me *how* to configure the per-service thresholds.

Once the thresholds are configured, you’ll see a green line for the warning threshold and a blue line for the critical one, like in this graph:

munin-it

Some of Munin’s plugins already have configured thresholds (such as disk space monitoring which will send a warning at 92% usage and a critical alert at 96% or so). But others don’t, and I wanted to keep an eye on e.g. system load, network throughtput and outgoing e-mail.

The mail command can be configured in /etc/munin-conf.d/alerts.conf:

contact.myname.command mail -s "Munin ${var:group} :: ${var:host}" thisisme@somewhere.com

Next in /etc/munin.conf, under the specific host I want to receive alerts for, I did something like:

[www.myserver.com]
    address 127.0.0.1
    use_node_name yes
    postfix_mailvolume.volume.warning 100000
    load.load.warning 1.0
    load.load.critical 5.0
    df._dev_sda1.warning 60

This will send alert if the postfix plugin’s volume surpasses 100k, if the load plugin’s load values surpass 1.0 or 5.0 (warning and critical, respectively) and if df plugin’s _dev_sda1 value is over 60% (this is disk usage).

Now here’s the tricky part: How to figure out what the plugin name is, and what the value from this plugin is? (if you get these wrong, you’ll get the dreaded UNKNOWN is UNKNOWN alert).

Just look in /etc/munin/plugins for the one that monitors the service you want alerts for. Then run it with munin-run, for example, for the memory plugin:

$ sudo munin-run memory 
slab.value 352796672
swap_cache.value 6959104
page_tables.value 8138752
vmalloc_used.value 102330368
apps.value 413986816
free.value 120274944
buffers.value 215904256
cached.value 4964200448
swap.value 28430336
committed.value 962179072
mapped.value 30339072
active.value 2746691584
inactive.value 2787188736

These are the values you have to use (so memory.active.warning 500000000 will alert if active memory goes about 5GB).

A tricky one is diskstats:

# munin-run diskstats
multigraph diskstats_latency
sda_avgwait.value 0.0317059353689672
sdb_avgwait.value 0.00127923627684964
sdc_avgwait.value 0.00235443037974684

multigraph diskstats_utilization
sda_util.value 6.8293650462148
sdb_util.value 0.000219587438166445
sdc_util.value 0.000150369658744413

In this case, use diskstats_utilization.sda_util.warning (so the value in “multigraph” is used as if it were the plugin name).

diskstats_utilization.sda_util.warning 60
Categories
English Geeky

Updating lxc image/container caches

One of lxc’s nice time-saving features is that, after initial container creation, it will cache the files it downloaded to do so, and when you create a new container using the same template/version/architecture, it will leverage the existing files and create the container with minimal downloads and really quickly.

A downside of this is that the cache can become stale; this is apparent when you want to install a package in a container and apt-get gives 404 errors indicating the version of the package the container knows about, is no longer available in the archive (most likely superseded by a new one).

This is easily fixed by always doing apt-get update in the container prior to any package installs/upgrades. However, it’s cumbersome, and if you’re creating dozens of new containers every day, the bandwidth and time spent re-downloading can quickly add up.

To update the “base image” or cache, which resides in /var/cache/lxc for each version, you can do two things.

most templates also support –flush-cache so if you’re calling lxc-create directly, just add an extra –flush-cache as template args (after –) and the cache will be flushed before making the container. Something like

sudo lxc-create -n ubuntu -t ubuntu -- -r trusty --flush-cache

this will obliterate the existing cache and re-download everything before creating the container.

If you want to update an existing cache do something like:

sudo chroot /var/cache/lxc/trusty/rootfs-amd64/
apt-get update
apt-get dist-upgrade
apt-get clean
exit

this will update the cache and all subsequently-created containers will know about the latest package versions.

 

Categories
English Geeky

WiFi interfaces on Ubuntu Server

Sometimes you may want to configure a wireless interface on a system with Ubuntu Server. The most common use case (for me, at least) is to run some tests with server, which require two network interfaces, on a laptop (it’s what I have available to play with) with an ethernet interface and a wireless interface. As long as Ubuntu sees the wireless interface, it’s quite easy to set things up so the wireless comes up at boot time.

You will probably need to set up the server to forward and masquerade the internal network (usually, the ethernet segment is the internal one, while the wireless counts as the “outside” interface). There are plenty of tutorials to do this over the internet, so I won’t extend this post by detailing that.

Of course, the wireless will grab a dynamic IP address, so use caution with that as the address may change (or, assign a static one from your router’s unused range). Anyway. Put this in /etc/network/interfaces:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 10.10.10.1
netmask 255.255.255.0

auto wlan0
iface wlan0 inet dhcp
wpa-ssid your-network-ssid
wpa-ap-scan 1
wpa-proto RSN
wpa-pairwise CCMP
wpa-group CCMP
wpa-key-mgmt WPA-PSK
wpa-psk your-network-password

Then you can do ifup wlan0 to bring the interface up. It should also come up automagically at boot time.

Categories
English Geeky Uncategorized

Speeding up local debian builds with sbuild (eatmydata, apt-cacher-ng and config laziness)

As part of the team that maintains several testing tools for Ubuntu, including checkbox, I sometimes find myself needing to build .deb packages from our source tree.

600px-Old_timer_structural_worker2
“building stuff is hard…”

A simple way of achieving this is of course to run dpkg-buildpackage or even bzr-buildpackage. Assuming all build-deps are correctly installed in the host system, this will result in a nicely built set of .debs.

This approach has a few caveats, in that it’s different from the build process actually employed to create the packages that ultimately get uploaded to Ubuntu (or even the ones available in Launchpad PPAs).

The two main differences are that Launchpad builds the packages in a “clean” environment, installing build-deps from scratch, whereas dpkg-buildpackage will rely on what’s installed in the system. So if you miss specifying a build-dep, your local build may work because you have it installed, but the PPA build will fail because it will not be present.

The second big difference is that with the local approach, you’re “limited” to building packages for the “host” system. Sure, you can specify a different target release in your debian/changelog file, but some aspect of your build may be tied to your system’s tools, versions and layout, and if for some reason they don’t match the actual target at installation time, things will fail in interesting ways.

Clearly, one way to test what the Launchpad build process will spit out is to build a source package and dput that to be built directly on a PPA. The problem here is that the feedback loop becomes excruciatingly slow; PPAs are a shared resource and build times can go from minutes to many hours.

Based on all this, it makes sense to try to use a local build environment that more closely replicates what PPAs do to build your packages.

Fortunately, the PPA builders use free software, so it’s relatively easy to do local builds in a similar environment, completing quickly due to use of local resources, and only upload to Launchpad once you’re pretty sure your build will succeed.

The software in question is sbuild, and I already wrote a post detailing how to install sbuild and set up a build environment for any Ubuntu release you need.

This setup worked fine for the occasional package build when you know packaging is mostly correct. For a fast build such as checkbox, setting up the build environment with all needed packages and build-deps takes about 10 minutes (depending mostly on download speed for all the packages). Of course on a more complex package, compilation time may start to be a factor.

Anyway, the 10-minute time can be too slow if you’re trying to fix a tricky problem and need a fast feedback loop. Plus the process produces a lot of transient files and downloads a set of packages many times, so there’s plenty of room for improvement here.

Speeding up local package installation and build

Eatmydata: it's so fast! (but not too safe)
Eatmydata: it’s so fast! (but not too safe)

A large part of the time spent doing the “local” part of the process is writing files to disk. One way to speed this up is to use a ramdisk to store the build. I’m too lazy and have too little RAM to use this approach, so the alternative was setting up eatmydata inside the chroot. Since these are mostly temporary files or throwaway packages, it’s OK to lose the safety of constant syncs in exchange for a huge boost in speed.

The setup for eatmydata inside the chroot is described here. This looks a bit hard to automate, but luckily we don’t have to, as recent versions of mk-sbuild simply support a –eatmydata parameter, if given this will install eatmydata inside the chroot and do the choot config file change to enable eatmydata.

Adding PPA

You can add a custom PPA to an image. Once the chroot image is built, enter the “golden master”:

sudo schroot -c source:saucy-amd64 -u root

You can add a deb line (get it from launchpad) to your sources:

cat >>/etc/apt/sources.list.d/something.list
 # Copy line here

Then you need to get the GPG key for the PPA and add it manually with the very basic tools provided in the chroot (sorry, no apt-add-repository):

apt-key add -
 # Paste GPG armored key here

Then exit the golden image. After this, your builds from this chroot will be able to fetch packages from the PPA.

Again, that’s a bit of work to do for each VM. Instead, what I did was create a file in /etc/schroot/setup.d to do this automatically. You can of course replace the PPAs you need in the echo lines at the end. Name the file something like 81add-ppas:

#!/bin/sh
set -e
. "$SETUP_DATA_DIR/common-data"
. "$SETUP_DATA_DIR/common-functions"
. "$SETUP_DATA_DIR/common-config"
echo "$STAGE" >>/tmp/stages
 if [ $STAGE = "setup-start" ] || [ $STAGE = "setup-recover" ]; then
 echo "APT::Get { AllowUnauthenticated "1"; };" > $CHROOT_PATH/etc/apt/apt.conf.d/80unauthenticate
 info "ADDING PPAS"
 SLD_PATH="${CHROOT_PATH}/etc/apt/sources.list.d/roadmr.list"
 . $CHROOT_PATH/etc/lsb-release
 MY_RELEASE=$DISTRIB_CODENAME
 [ -n "$MY_RELEASE" ] || MY_RELEASE=trusty
 echo "# Added by the schroot setup mechanism (roadmr)" > $SLD_PATH
 echo "deb http://ppa.launchpad.net/checkbox-dev/ppa/ubuntu $MY_RELEASE main" >> $SLD_PATH
 echo "deb http://ppa.launchpad.net/ubuntu-sdk-team/ppa/ubuntu $MY_RELEASE main" >> $SLD_PATH
 fi

Notice that again, I was very lazy and instead of downloading the gpg keys as shown above (as for some reason trying to run gpg from the setup script didn’t work), I just configured apt to allow unauthenticated packages. Since this sbuild is mainly for testing purposes it’s not a big deal to skip this verification step. Also, there’s some logic to automatically detect the chroot release, so the same config file works equally well for any Ubuntu release.

Apt-cacher-ng

As the name suggests, this nifty utility will cache packages so the next time you need them they’ll be fetched from local storage rather than from the network. A bit of config is needed to have sbuild download packages from here.

Hello, I got these packages cached for you...
Hello, I got these packages cached for you…

First, install apt-cacher-ng on the host system. You can verify it’s listening on port 3142 by any means you like.

Then, to set it up automatically in chroots, add this to the host system’s  /etc/schroot/setup.d/80apt-cacher-ng (rather, create that file; it doesn’t exist by default):

#!/bin/sh
 set -e
 . "$SETUP_DATA_DIR/common-data"
 . "$SETUP_DATA_DIR/common-functions"
 . "$SETUP_DATA_DIR/common-config"
 if [ $STAGE = "setup-start" ] || [ $STAGE = "setup-recover" ]; then
 echo "# Added by the schroot setup mechanism (roadmr)" > "${CHROOT_PATH}/etc/apt/apt.conf.d/80proxy"
 echo "Acquire::http::Proxy \"http://127.0.0.1:3142\";" >> "${CHROOT_PATH}/etc/apt/apt.conf.d/80proxy"
fi

With these two setup.d scripts and the –eatmydata magic, it’s easy to create sbuild environments which will be much faster when building packages.

As a comparison, building msmtp (chosen because this tests mainly the speedup components, not needing any packages from a PPA) takes about 40 seconds with these suggested tweaks:

Build Architecture: amd64
 Build-Space: 5948
 Build-Time: 17
 Distribution: trusty
 Host Architecture: amd64
 Install-Time: 12
 Job: msmtp_1.4.31-1.dsc
 Machine Architecture: amd64
 Package: msmtp
 Package-Time: 40
 Source-Version: 1.4.31-1
 Space: 5948
 Status: successful
 Version: 1.4.31-1
 ─────────────────────────────────────────────────
 Finished at 20140320-1301
 Build needed 00:00:40, 5948k disc space

Whereas on a non-tweaked chroot it takes about 1:38 minutes:

Build Architecture: amd64
 Build-Space: 5568
 Build-Time: 17
 Distribution: trusty
 Host Architecture: amd64
 Install-Time: 31
 Job: msmtp_1.4.31-1.dsc
 Machine Architecture: amd64
 Package: msmtp
 Package-Time: 98
 Source-Version: 1.4.31-1
 Space: 5568
 Status: successful
 Version: 1.4.31-1
──────────────────────────────────────────────────Finished at 20140320-1310
Build needed 00:01:38, 5568k disc space

It looks like they’re about 3 times faster, but that’s misleading because I deliberately chose a small, quick-to-compile package. Still, you can at least reduce network and disk access very easily now. Note, also, that my test system has a fast SSD. Speedup on a traditional rotary magnetic hard-disk is likely to be much higher.