Turn Teamwork in to a Devops command center

We use Teamwork every day to manage our software development work for our clients. I am always interested in pushing Teamwork a little further to see if we can perform more and more tasks within the application so that it is transparent and accessible to our clients, co-workers…and save some time too!

In a previous post, I wrote about using Teamwork as an email support system and using a 3rd party API to perform sentiment analysis to help prioritise tasks. That was just the first step.

A lot of my day to day work goes a little bit beyond software development and into infrastructure, or ‘devops’, especially at the beginning of a new project. I wanted to see if I could perform a few infrastructure related tasks from within Teamwork, to automate the process a little more and even allow my co-workers to perform these actions too even if they are not comfortable with server environments, DNS Settings, SSH access and so on.

Some of the routine tasks that fall under the ‘devops’ umbrella which I can automate and trigger from within Teamwork include registering domain names, using Amazon’s Route53 for managing the nameservers/DNS for the domain, setting up sub domains and pointing a (sub) domain to an appropriate IP address on a server.

The general workflow is:

When a new task is added to Teamwork, a ‘hook’ POST’s the details of the task to a PHP script I have created.

This PHP script parses the task details and looks out for particular keywords to perform actions on in the task text. I have it set up to look for the following commands:

(I have prefixed the commands with ‘murbot’, short for ‘murrion robot’, mainly for fun but also to make sure to differentiate it from a regular task, just in case a client adds some text to a task and accidentally registers an odd domain name!).

  • murbot: register domain domain.com (a Teamwork hook calls a remote PHP script and the chosen domain is purchased (if possible) with namecheap.com, thanks to their very easy to use API. ) (The domain is set up with Murrion contact/technical/billing details)
  • murbot: use route53 for domain.com (a hook calls the PHP script and a hosted zone is created on Amazon Route53. 4 Nameservers are provided by Route53 and those are used to update NameCheap.com, so it knows to use Route53 for future DNS settings)
  • murbot: create recordset for 192.168.1.2 (a hook calls the PHP script again and a RecordSet is created on Amazon’s route 53 to point the domain name to the IP address of the server when the web application will reside.
Perform actions in Teamwork

Perform actions in Teamwork with murbot

At this point, the necessary domain/DNS stuff is created for a new project in seconds. Any results from the API calls are added as comments to the task and the task is closed.

There is more to do on the server side, such as creating virtual hosts, folders, permissions, users, databases, but that’s for another blog post!

Some other actions I am working on include:

  • Murbot: use ses for domain.com (this would create the appropriate TXT and DKIM CNAME DNS records in Route53 to use the domain name with Amazon’s Simple Email Service)
  • Murbot: create subdomain test.domain.com for 192.168.1.2 (this would add the appropriate A name DNS record in Route53 and point the sub domain to the appropriate IP address)

Of course, each of these commands needs to be learned by anyone on the team that wants to use them, but it’s a lot quicker and safer to learn and action these items from the safety of Teamwork, rather than logging in and out of different services.

All of this is possible thanks to:

 

 

 

Teamwork as a support system with automatic prioritization of tasks

I signed up for a 15 day trial of HelpScout recently to try it as a central Support system here at Murrion.

It allowed us to create a central inbox for all the members of our team to see incoming support emails from our clients.

This is ideal if a client emails me directly and I don’t get to the email in a timely manner, one of the other members of the team working on the same project can see and respond to the email faster than I can.

We used it during the trial period with a small number of willing clients and overall found it very useful, we were able to communicate back and forth and assign and label support requests, visible to all team members.

When the trial ended, I couldn’t bring myself to pay an additional sum each month, despite its low cost. There is also a Free account, but that is limited to 3 users and we would have 5 or more. Overall it was useful but we ended up having 2 systems to check, Teamwork and Helpscout and my goal is to improve workflow and not add more unnecessary moving parts.

I wanted to continue to use Teamwork which is already such an integral part of our everyday workflow.

So I decided to continue to use the new Support email address and route the incoming emails to Teamwork instead of Helpscout.

Teamwork supports creating tasks and other items via email, but I wanted a little more flexibility. I want incoming support emails to route to the correct project, to rank in low, medium or high priority and to assign to the correct person, all automatically.

So over the weekend, I put the following in place, Turning Teamwork in to a Support system with automatic prioritization of tasks.

I used a couple of additional tools to accomplish this and its worth noting that all of these are 100% free if you wanted to replicate this.

Prioritized support task in Teamwork

Prioritized support task in Teamwork

The general workflow is:

  1. An existing client can email us using support at murrion dot com
  2. Any email sent to this address is received by Mailgun which Posts the data back to our server.
  3. The content of the email is passed through a Lymbix sentiment analysis tool.
  4. Depending on the sender of the Support request, the relevant Project to connect to in Teamwork is discovered.
  5. Using the Teamwork API, the support details are added to a list in Teamwork. The list is created if it doesn’t already exist.

Based on the sentiment analysis performed in step 3 above, the Support item is labelled as a High or Low priority in the task list.

If items are deemed high priority, we are notified via email from Teamwork. If its low priority, it just joins the list quietly.

It works so well I might route all of my emails in to Teamwork every day and have this system prioritize my day to day tasks!

What I learned from BulletHQs development team

I worked with Bullet HQ’s API recently so that I could import account and related data in to their system. Afterwards I had the opportunity to talk with John Farrelly, responsible for Bullet’s Product Development about how they develop their software.

BulletHQ is an excellent Accounts and Payroll web application, based in Dublin. They recently began offering their Accounts package for free to individuals and businesses with an optional paid-for Payroll section.

How is BulletHQ developed?

I asked John how they develop their software, as an accounts package that businesses rely on, it needs to be rock solid.

I was delighted to hear that Bullet believe in and actively use continuous integration practices in developing their software, it is an approach to development we are actively moving towards here too in Murrion.

Continuous Integration (CI) is a practice of automating software builds using tools such as Travis, testing the software and then deploying Live code changes frequently, sometimes several times per day.

The development team spend a large amount of their time in writing tests and are big proponents of Test Driven Development (TDD), meaning writing tests first and developing code until those tests pass.

The team have so many tests in place for their software that it can take several hours to run the full suite!

To organise their work, the team use Sprints, often 2-weeks long. Sprints are an Agile software development approach. Sprints are a period of time to focus on one or more tasks. In team meetings, they collectively decide upon and take responsibility for the tasks that need to be included in the Sprint period.

What’s under the hood?

Bullet is built using Java. I was saddened to hear it’s not developed using PHP :)

What’s next for Bullet?

Bullet are putting the final touches to a new Support section on their site, to help new and existing users of their software to get answers from staff and other members of the community.

On a development front, they are focussed on choosing the right platform for the documentation of their API, so if you’re a developer, watch this space!

Introducing MouseJS for visualising user activity on your website

Mousejs.com is a new site we put together to help some of our clients to gain some insights about user movements on their own website or web application.

It works by tracking the precise movements of a user’s mouse cursor while they are navigating a webpage. Once the user has left the page, a screenshot of the page is then created and the user’s mouse movements are ‘drawn’ or super-imposed on to the same page to show a visual representation of the users movements.

It works by placing a very simple Javascript snippet on any page(s) of your site that you want to track. You can then log in to the Mousejs.com site at any time and view all of the images that Mouse has generated for your site.

Any images that are created are available almost straight away. It just takes a few seconds to generate the screenshots and draw on them before they are ready to view.

Some scenarios where this is useful

  • Spot potential spammers and plagiarists! – I didn’t think of this scenario at all until I saw it happen on this site. I could see some images of a user visiting our homepage and highlighting our contact email address, presumably copying & pasting it to send me an email at some point, or spam it!
  • Remove old content – Over time, it’s possible to see images, buttons or areas on your site or app that users seem to have no interest in and never go near or click upon and therefore might be worth removing or updating with new content.
  • Test new content – This is very similar to the previous point. If you have added new buttons, links or images to your site, you can visually see if uses are moving towards it or clicking on it.

Some current weaknesses in this service

MouseJS is very good at recording the precise movements of a user’s mouse cursor. However, if a website is particularly dynamic with content appearing/disappearing or moving based on users movements or clicks, then the next step that MouseJS takes of generating a screenshot will not capture this visual change.

It currently assumes that a page doesn’t change and it just takes one screenshot of that page per user per visit, therefore not giving a full visual representation of the user’s interactivity with that page.

A good example of this is if a modal window popped up and the user had to press a button to clear it away, then in the screenshot that MouseJS generates it would look like the user clicked and moved around the centre of the page for no reason, it would not have captured the pop up modal window.

However, we’re working on this and the plan is to have MouseJS take multiple screen shots of a page during a visit to replicate any changes and overlay movements on an animated image rather than a static image.

Some features that will be available in the near future

  • Heat maps – by combining several images together, the areas of a page that are used the most are highlighted.
  • Visualise users over multiple pages – At the moment, it’s possible to track users on all pages of your site, assuming the Javascript snippet of code is embedded in every page. However, when visualising the screenshots, the images are shown individually and with no context. One of our next steps will be to group images together so that all the pages that a user visited on your site are grouped together and maybe even animated too.

Under the hood

This service was built using some Javascript for recording mouse movements and some PHP to receive and record the mouse movement data. Then, PhantomJS, a headless browser is used to generate the screenshot and Codeigniter is used to give the user an Admin panel to log in and view the screenshots.

Start testing your API with CodeCeption

This is a brief step by step guide to install CodeCeption on your local machine and to create a basic but useful test for a fictional ‘Categories’ API endpoint.

Codeception is a Testing framework that you can use for creating Unit Tests, Acceptance tests and Web Services/API tests too. For more information, http://codeception.com/

The test created below makes sure an API endpoint such as http://api.domain.com/categories is online and responding as expected. It tests for a HTTP 200 code in the response, that the response is in a JSON format and that the response contains ‘success’.

This same test could be recreated for many other endpoints in your API. Also, If you store your code using Github or Bitbucket, you could use a service such as http://codeship.io or https://circleci.com/ to run those tests automatically any time you commit a change to your code to let you know that all of your API endpoints are online and responding as expected and only deploy changes Live if all your tests pass.

  1. Download codecept.phar from http://codeception.com/ and place it in the root of your project
  2. In the terminal, in the root folder, write ‘php codecept.phar bootstrap’, this creates the Tests folder and other bootstrap related files
  3. In the terminal, write ‘php codecept.phar generate:suite api ApiGuy’, this creates the /test/api folder where we will keep our API tests.
  4. Update the existing file called /tests/api.suite.yml with the code written below to give CodeCeption some basic information about our API, replacing ‘domain.com’ with your own URLs.
  5. In the terminal, write ‘php codecept.phar build’ to build the classes that we will need
  6. In the terminal, wite ‘php codecept.phar generate:cept api Categories’ to create a ‘Categories’ test in /tests/api/CategoriesCept.php
  7. Update CategoriesCept.php with the code below to tell it what to test.
  8. Finally, In the terminal, write ‘php codecept.phar run’ to run your tests!

api.suite.yml

class_name: ApiGuy
modules:
      enabled: [PhpBrowser, REST, ApiHelper]
      config:
            PhpBrowser:
                  url: http://domain.com/
            REST:
                  url: http://api.domain.com/

CategoriesCept.php

<?php
$I = new ApiGuy($scenario);
$I->wantTo(‘Return all categories’);
$I->sendGet(‘categories’);
$I->seeResponseCodeIs(200);
$I->seeResponseIsJson();
$I->seeResponseContains(“success”);
?>