Cross-timezone date calculations using the “date”command

Working remotely for a timezone-distributed company poses an interesting challenge: that of having to figure out dates and times for people in different timezones. This involves not only the relatively trivial “what time is it now in A_FARAWAY_PLACE”, but “what time, in FARAWAY_PLACE_X, will it be in FARAWAY_PLACE_Z” and other fun things.

There are a handful of websites that have handy tools to do these conversions for you; but a problem I’ve found is that the web is going to the crapper, and these sites often have confusing UIs concocted by some javascript-crazed, CSS-infected webmonkey; and often they are completely swamped and rendered unusable by a rising tide of ads and other aggressive content (oh and some won’t let you do anything until you agree to them storing information in cookies in your browser – which they then bafflingly don’t use to store the *PREFERENCE* you have selected , so like a forgetful vampire, they ask you every single time if you want to accept their silly cookies).

I’ve known how to use the “date” command to show the date on a different place/timezone, which is already a huge timesaver:

but – today I was trying to answer “what time in TZ=”America/Chicago” is 1 PM, on Tuesday, in “UK/London“. This is interesting because it’s conversion between two timezones which are not the one I’m in, of a date/time in the future. So I was checking date’s man page for “how to convert a specific point in time”, when I realized date can do this for you! Right in the man page there’s this example:

Show the local time for 9AM next Friday on the west coast of the US

so then I combined that with the earlier one to come up with:

This combines

  • TZ argument to calculate dates for a specific timezone, not the current one
  • –date parameter to “display time described by STRING, not ‘now'”
  • Descriptive time specifications (1:00 PM next Tuesday – this is a pseudo-human-readable format which is not entirely intuitive – info date has the specifics)
  • TZ support *inside* the descriptive specification

And a list of known timezones can be obtained with timedatectl list-timezones.

SAML development tools

If you work on a system that needs to authenticate against an external identity provider (IdP), SAML is almost certainly a fact of life. Working on an actual Identity Provider, sometimes the concern is flipped and you need to ensure Service Providers (SPs) can authenticate against your IdP.

I inherited the somewhat clunky django-saml2-idp along with other developers in my team, and we’ve been maintaining it to add new features. If we were doing this today, we’d probably integrate the very complete OneLogin SAML library instead.

Developing with and for our somewhat homegrown SAML library is made easier by a set of developer tools. For example, OneLogin provides a toolbox to slice and dice SAML assertions; you can verify your assertions, extract attributes, see some examples, play with zipping and encoding, all in one place.

Once you have your IdP mostly working, it’s great to have a test SP to connect to it. For this, I’ve used the RSA SAML test Service Provider. You give some details about your IdP, and it will give you a URL that forwards you to the IdP for authentication, then back to the SP, which verifies authentication worked as expected and even shows you the attribute and auth payload received from the SP.

Once you get things mostly working but need to fine-tune or tweak something (I can never tell between issuer, ACS_URL and audience), the Firefox SAML Tracer extension is absolutely essential. It shows you all requests and responses, which ones contain SAML payloads, and lets you see the actual, decoded and formatted XML payload which makes it a breeze to troubleshoot.

There is an equivalent SAML tracer extension for Chrome but 1) Chrome is crap and 2) the Chrome SAML extension is crap. Use Firefox instead.

Remote display of a KVM virtual machine

In this case I’m hosting the VM on a fast server and trying to access the display on another system (a laptop).

One way to do it is by simply SSHing with X forwarding and running KVM like so:

This by default uses a terminal window, but it’s quite slow.

Another option is to start the KVM machine in nographic mode and enable a VNC server:

then on the desktop system use a vnc client to connect to the magic port:

KVM bridged to the LAN with DHCP

The goal here is to instantiate VMs with a br0 interface grabbing an IP from the LAN DHCP, so in turn the VM can instantiate LXD containers whose IP is also exposed to the LAN. That way everything is visible on the same network segment and this makes some experimentation easier.

Host configuration

Some info taken from this URL.

The metal host is running Ubuntu 18.04, which uses netplan. Here’s the netplan.yaml file:

With this, on boot the system grabs an address from the network’s DHCP service (from my home router) and puts it on the br0 interface (which bridges enp7s0, a Gigabit Ethernet port).

The system also has avahi-daemon installed so I can ssh the-server.local easily.

VM configuration

Next, the VM which I created using uvt-kvm:

The setup_network.sh script takes care of setting up the network 🙂 This can more cleanly be done with cloud-init but I’m lazy and wanted something fast.

The script deletes the cloudconfig-created .cfg file, tells cloud-init to NOT reconfigure the network, and drops the config file I actually need in place.

 

The network config in /etc/network/interfaces.d/1-bridge.cfg should look like:

LXD configuration

Finally,  install lxd. When asked to configure the lxd bridge, respond “no”, and on the next question you’ll be asked whether to supply an existing bridge. Respond “yes” and specify “br0”.

Now, when an lxd container is instantiated, it’ll by default appear on the same network (the home network!) as the VM and the main host, getting its DHCP from the home router.

When things break

Suddenly the bridge interface stopped working. I checked this to help diagnose it. But that wasn’t it. Turns out, I’d installed Docker on the main host and Docker messes with the firewall configuration by setting iptables -P FORWARD DROP. I just set it back to ACCEPT to get it working.

Buying a house in Montreal – Moving and Renovations

Now that you own the home, comes the time for those renovations from your loan. This means that we couldn’t move right away, since it was much easier for the contractor to work on an empty house.

This all went well, since we still had some time left in our apartment’s lease, so we were able to wait for most of the reno work to be done. But it makes sense for  you to check frequently on the work, particularly since the contractor is bound to have some questions and clarifications, choices need to be made, some materials need buying, and so on.

The not-so-nice part was that, as discussed in the earlier post, we had to pay the contractor out-of-pocket, because the portion of the mortgage loan covering the renovations isn’t released (the notary holds that money) until the renovation is entirely finished. But we had planned for this so we were able to pay the contractor while not breaking the bank or going into massive credit card debt.

My advice here is to tell the contractor about the loan conditions and ask him to finish the work as soon as possible. Once the work is complete, call your mortgage broker who will arrange an inspection with the lender. The lender will hire an appraiser who will come to your home and do the inspection, and they will absolutely insist on everything being entirely finished; no missing paint job, no doorframes without doors, no excuses.

On the bright side, the process is relatively quick, it was less than a week between calling our mortgage broker and receiving the notary’s call to pick up the cheque for the remaining money.

Then we were able to move in, but that, and the ongoing maintenance work needed by a somewhat aged house (but with lots of character and charm) are beyond the scope of this series.

Our timetable looked like this in the end:

Initial visit to the mortgage broker – February 10th

Initial contact with real estate agent – February 15th

Started house visits – February 20th

Made an offer on the house we liked – March 2nd

Offer accepted – March 6th

Closing day – April 26th

Possession date – May 10th

In the end, from the initial visit to the broker to get the loan preapproval to taking possession of the property, even including the weird 2-week period after the closing date, the entire process took exactly 3 months.

Buying a House in Montreal – Closing

Once the conditions have been fulfiled and the notary has all the documents, all that’s left to do is wait for “closing day”.

Before closing day, you’ll probably have an appointment with the notary to review all the documents and sign most of them. This includes the deed of sale, the mortgage agreement, and some other agreements.

Also before closing day, the notary is likely to request you give him the cheque for the downpayment. This has to be a certified cheque or money draft, so be sure to ask the notary and get the cheque well in advance.

My notary told me he likes to do things this way since it means a shorter appointment on the actual closing day, but some notaries may choose to do everything on closing day, which will mean about an hour at their office.

Finally, on closing day, the sellers and buyers meet at the notary’s office, the essentials of the purchase agreement and deed of sale are read and confirmed by all parties, money exchanged (the notary gives the sellers a cheque for the full amount of the purchase, this is made up of your downpayment plus the money the mortgage lender deposited in the notary’s account). Then signatures made (at this point we signed the deed of sale, meaning we own the house), and that’s pretty much it. You walk out of the ceremony being the owner, on paper, of the property you chose.

It may be that you’ll get the keys to the house during the closing; in our case, we had to wait an extra 2 weeks for the sellers to vacate the house (we all agreed to this in the promise to purchase), after which we met them at the house and received the keys.

Congratulations, you’re now a homeowner!

Buying a house in Montreal – Fulfilling the conditions of the offer

Our offer to purchase contained two main conditions: an inspection within 7 days of the acceptance, and obtaining an approved mortgage loan within 15 days of the acceptance.

Since we’re going for a renovation and extending the mortgage to cover that cost, it’s usual to add another condition to allow for a contractor to visit the space to provide the required work quote. In our case, we rolled the contractor visit the same day we did the inspection, so there was no extra visit required. The inspection had to be performed within 7 days of the offer’s acceptance, and we did it on day 5. We got the quote for the renovations on day 8. We sent all the documents for the mortgage beginning on day 1, completing them on day 8 with the renovation quote.

The lender sent us two “letters of commitment” to sign. The first one about 4 days after we had the accepted offer, then a second one about 3 days after sending the renovation quote (so 11 days into the process). We signed and returned them as soon as we received them, and then the lender sent the final approval letter 2 days before our 15-day deadline for completing financing was up, which was a bit nerve-racking. So my advice would be:

1- Do the inspection as soon as possible. If possible, within 3 days of the offer acceptance.

2- Fulfil all of the lender’s requirements as soon as possible. Don’t drag your feet; get a notary well in advance.

3- Ask for a slightly longer time to get financing. We asked for 15 days because of the extra renovation quote time required.

Our agent and mortgage broker were very helpful during all this process and mostly we only had to sign documents and collect some information. Most of this had already been pre-screened by the mortgage broker so in reality, things went pretty smoothly.

Once you get the letter from the lender or mortgage broker, send it to your agent (our broker actually sent the letter to the agent for us – mortgage brokers are really helpful), this should fulfill all the conditions of the offer.

Most of the next steps will be done by the real estate agents, the lender/bank and the notary. At this point I did call everyone to ask what I had to do next, and the main response was “nothing” and “wait for the notary to call you”.

Buying a house in Montreal – Making an offer

After visiting 4 houses we decided on making an offer on the fourth one: it’s well-located, has a large front yard (and no backyard!), and two levels, the second of which has a very weird layout.

Apparently the no-backyard thing is a deal breaker for most people, but it wasn’t for us because the front yard is a pretty good size. The second odd thing is the layout; this may have put a lot of people off, but our realtor suggested it could easily be remodelled into something more traditional and fitting to our needs.

So after coming home and sleeping on it, we decided we could renovate the space before moving in, and that would make the house good for our needs. So then you call your agent and tell them you want to make an offer to purchase the house.

The offer is called “promise to purchase” in Québec, and it’s a very standard format, but it made a lot of sense for the agent to help us with that. She asked for a beginning price we’d be willing to offer, and offered advice on whether to modify this (she actually suggested we go lower than our initial idea, so we went with her advice).  She’ll also help decide which conditions to add to the offer. In our case, it was conditional to getting the financing, and on having an inspection performed on the property.

The seller can either reject your offer, accept it as-is, or send you a counter-offer. In our case we got a counter-offer, with a higher price (but still below what I’d wanted to offer in the first place – this validated our agent’s criteria about the initial offer). Since this was well within what we were expecting, we accepted the counteroffer. This starts the clock ticking to fulfil the conditions, the main ones being financing and the inspection. Talk to your agent and ensure you allow enough time for all this to happen.

 

Buying a house in Montreal – Visiting houses

Once we identified a few houses we might be interested in, we asked our Realtor if we could visit them.

First we sat together and looked through the pictures, checking the descriptions and the locations on the map. Since location was one of our main criteria, we discarded a bunch of houses that, while in our desired neighbourhoods, were too far from places of interest. Some others look terrible in the pictures; a few were discarded simply because the pictures showed them to be in bad condition, meaning they’d need lots of renovations before being usable.

We divided the remaining houses into “top picks” and a secondary list of “maybe”s which we didn’t outright discard. But we decided to focus first on the top picks.

Depending on how active the market is, some of the houses may be sold before you get a chance to see them. Three of our top picks were in this situation, narrowing down our list.

We send the listings to our realtor who then arranges for the visit which is done in the presence of your realtor; the house may be shown by the current inhabitants or by the selling agent. Having the realtor there was a good advantage because she pointed out many aspects we could have overlooked: potential for expansion, whether the doors/windows look new, any superficial evidence of damage or poor construction, things that may need changing or considering, and so on.

Once you’ve visited a few houses and formed a mental picture of what you liked and disliked about them, you can think about whether any of those are convenient for you. After seeing them, you have all the facts (and the subjective opinion you may have on each of them) and can do an analysis to see if any of those satisfy your criteria. If none do, then just keep waiting! One thing we observed is that new listings pop up almost daily (and the good ones are snapped up quickly), so don’t feel too rushed if you don’t find a house you like among current candidates. The houses you do see will allow you to clarify your needs and desires, so that when the right one comes along, you’ll be ready to move quickly.

We looked at 4 houses:

    One had a very nice garden and lots of land and potential, but had a lot of moisture and foundation problems and was really really far.
    Another was move-in ready and had a very convenient layout, as well as a dog-friendly and big garden, but it was at the very top of our budget plus was quite far from most of the amenities we like.
    A third one was beautifully renovated and move-in ready, and was quite close to transit and favoured amenities, but it had no room for growth as it had only a small front yard.
    A fourth one was relatively well-kept and renovated,  close to amenities and transit and while it had only a front yard, it was quite large and left a bit more room for growth. But the spaces were very untraditional and tailored to the specific needs of the inhabitants (which didn’t necessarily match our needs).

As you can see, all houses have their pros and cons. If you find one that has no cons and is within your budget, by all means go for it! But more often, it’ll be a game of compromise and deciding which aspects are more important to you and which ones can be overlooked or postponed.

Remember there are two things you can’t change about the house: the location, and the plot of land it sits on.

There’s one thing that’s difficult to change: the general structure or “bones” of the house (although if sufficiently motivated, you can always tear it down and build anew).

Most everything else can be tailored to your needs, given enough time and money.

 

Bisecting Python unit test errors to find test interdependencies

 

Many of our test runs use parallelization to run faster. Sometimes we see test
failures which we can’t reproduce locally, because locally we usually run
sequentially; and even then, the test ordering seems to be somewhat
unpredictable so it’s hard to reproduce the exact test ordering seen in our
test runner.

Most of the time these failures are due to unidentified test interdependencies:
either test A causes test B to pass (where running test B in isolation would
fail), or test A causes B to fail (where running B in isolation would pass). And we have seen more complex scenarios where C passes, A-B-C passes, but A-C fails (because A sets C up for failure, while B would set C up for success). We added some diagnostic output to our test runner so it would show exactly the list of tests each process runs. This way we can copy the list and run it locally, which usually reproduces the failure.

But we needed a tool to then determine exactly which of the tests preceding the failing one was setting up the failure conditions. So I wrote this simple bisecter script, which expects a list of test names, which must contain the faily test “A”, and of course, the name of the faily test “A”. It looks for “A” in the list and will use bisection to determine which of the tests preceding “A” is causing the failure.

As an example, I used it to find a test failure in Ubuntu SSO:

 

 

Buying a house in Montreal – Purchase plus Renovations Loan

We were initially reluctant to consider non-move-in-ready properties, reasoning that we preferred to devote money to a down payment and not have to contend with either moving delays or  construction crews around the house. But if the occupancy dates allow for it, considering a renovation job may be worth it, because it could allow you to snatch a more affordable property (perhaps one that buyers are overlooking because of quirky layouts or being in bad condition, needing a kitchen or bathroom renovation, and so on) and use the saved money to renovate, which instantly raises the property’s value.

If you’re concerned that this will leave you somewhat cash-strapped, there’s an option called “Purchase plus Improvements mortgage”. Essentially it allows you to borrow an extra amount (up to 10% of the value the home will have after the improvements) which will be added to the mortgage.

The rules are somewhat complex, particularly the part where you need a quote from a contractor before finalizing the purchase and loan, and the fact that the money is not advanced to you, but paid after the work is done and an appraiser has verified it’s done as requested, meaning you may need to pay for it upfront – it does defeat the purpose a bit, but consider that you could pay for this work with consumer credit or a line of credit, repay that once the extra mortgage amount is released, and then benefit from the very low mortgage interest rates.

In any case, it’s an option which opens up a good set of properties you may pass over because they need renovation.

I suggest googling for “Purchase plus Improvements mortgage” for more information on the rules, process and limitations.

Buying a house in Montreal – the credit report

A few of the resources I read mentioned it was a good idea to get a credit report prior to visiting the mortgage lender/broker, so any inaccuracies can be corrected in advance. They indicate the credit report can be obtained free of charge through Equifax or TransUnion.

I went to Equifax (note this was before their 2017 breach – I obviously now recommend you go to TransUnion instead). And their home page is a bit scary offering products protecting you from identity theft (I guess that didn’t help or protect those affected in 2017). But I was only interested in the credit report and score (the score is not necessary but it’s a nice single-number summary of your credit and I though it worth it). They have an option to purchase on-line for $23 so I went with that.

But remember the report can be obtained for free? understandably, this is not terribly visible in their front page, but there it is: “You can receive a free credit file disclosure from Equifax Canada Co. via Canada Post“. That link will take you to a form you can fill out and mail or fax (Fax, really?). So for the cost of a stamp and a bit of waiting you can also have your free credit report.

I was impatient and paid the $23, only to get a scary error when trying to get the report, which necessitated calling Equifax, in the course of the call they tried to upsell me on their credit report monitoring service (it’s cheaper, they said, only $16/month instead of the $23 you’re paying – conveniently not mentioning than the $23 is a one-shot charge). Which product you choose is up to you, just remember to stand your ground if you’re not interested in the more expensive one.

The credit report indicated a reasonably high score and no unusual activity, and should look good to any prospective lenders, so this phase is complete and we’re good to go!

Buying a house in Montreal – the stress test

One of the changes to mortgage rules for 2016 in Canada is the creation of the “stress test“, meant to cool down hot real estate markets and keep people from buying houses that stretch their financial capabilities.

If you’re going for a high-ratio (less than 20% down payment) mortgage, lenders are required by law to check your payment capacity as if your loan interest rate were as high as the standard five-year rate (which currently is 4.94%), even if your eventual mortgage will actually be at a much lower (currently around 2-2.5%) rate.

The FCAC calculator makes it very easy to check what your maximum loan will be, once this rule is taken into account. Just enter your information and your expected interest rate to calculate your real maximum mortgage. Next, change the interest rate to 4.95% (I went super safe and put in 5%). It will tell you you won’t qualify, but you can now play with the maximum property value until it shows you you’re likely to be approved.

In my case, it resulted in a reduction of 18% in the maximum price I could afford, which is not terrible because all my previous calculations were taking this into account. Some people may be surprised, and discouraged out of the house hunting process by this, but if you know about this rule and factor it in your calculations prior to starting the process, you’ll know what to expect and how to compensate (get more money, save up for a larger down payment, lower your house price range).

 

Buying a house in Montreal – where to start?

So we decided to buy a house, what will the journey look like?

There are plenty of easily-googlable resources on the house buying process in Canada and in Québec more specifically (here’s the two most detailed I’ve seen: FCAC and CMHC), so I won’t try to repeat that information, but I’ll document the specifics of our process which will invariably result in a brief outline of the steps to follow.

Roughly what we’ll try to do:

  1. Get a relatively good family income so we can qualify for a reasonable mortgage loan.
  2. Build up a credit history.
  3. Save up for a down payment.

We’ve worked on those first three steps since we moved to Canada: I’ve been fortunate enough to have a stable and well-paid job, which has allowed us to use consumer credit responsibly, so should have a pretty good rating. It also allowed us to save for a down payment. So at this point we should be ready for the next parts of the process:

  1. See a lender to get financially checked and pre-approved for a loan. You can go for a well-known financial institution, perhaps your bank, or you can go to a mortgage broker, which is what I’m planning on doing.
  2. Once you know your price range, you can start looking at houses in your desired areas.

BUT before you can start with this, you should know roughly how much you can afford, be realistic with your inputs and use one of the available online calculators. I like this one which will tell you how much you should be able to afford, and this one which calculates your estimated payments. And this one is very simple but also very detailed as to the criteria used to estimate affordability. It makes sense to use this so you’re not disappointed when the broker tells you you can only afford a tiny shack in the boondocks :).

You should also have a pretty good idea of whether you like your target neighbourhood. Montreal is a geographically large city and neighbourhoods can differ, so it makes sense to check the ones you like and make a short list. If you don’t care where you buy, there’s something for almost any price range, but I don’t think that’s very common.

A possible problem with the neighbourhood you like is whether you can afford it. If you can’t just yet, there are two options: choose a different one or get more money (higher salary, larger down payment).

Once I identified our target neighbourhoods, I started scouring centris.ca frequently, looking for houses in (and out of) our price range, checking their pictures and prices, nearby amenities, and comparing several possible neighbourhoods. We ended up discarding one of those, even though it was cheaper and had more inventory, because we decided we didn’t really like it that much. So we’re focusing on one of the other candidates, and also looking at adjacent neighbourhoods, which can be cheaper while still being closer to the amenities we want.

OK, so knowing how much we can afford (per the calculators) having located (and lived in) a neighborhood we like and knowing the approximate price range for homes here, and knowing it is within our affordability, I’m ready to hit the mortgage broker.

Resources:

http://www.fcac-acfc.gc.ca/Eng/resources/publications/mortgages/Pages/home-accueil.aspx

Ansible task that retries

The task can be whatever you want: I used uri but it’s more usually shell or something like that. The main thing is that you need to use register so you’ll have something to check in the until: condition.