I’ve made another thing (it’s been quite a while judging by the last post)! Over the weekend, I set up a script that periodically checks and emails me if tickets have become available on a certain website for the date that I’m interested in. It’s basically some ruby code that uses Watir to analyse a webpage, Mailgun to send me an email and is hosted on Heroku with the scheduler plugin.

For anyone living in London, it’s an unavoidable fact that you have to book tickets for anything good way in advance. This has bitten me far too many times in the past. With my sister arriving in a month’s time and all bars and restaurants at the Sky Garden being booked out, I was finally motivated to do something about it. Sky Garden have a number of free tickets that they make available 3 weeks in advance. So rather than me having to click on the site every single day to check for tickets, I set out to find a more programmatic solution…

First I had a look in the Chrome web console and the Tamper plug-in, which allows you to intercept HTTP requests. The ticket availability on the website is presented in a table, with the individual days being individual clickable cells. Clicking on an individual date showed requests to a specific booking related API. However curling the API directly returned permissions errors. I assumed that this had to do with some sort of fancy session set up and I didn’t have the time or dedication to figure out how that worked. Instead I realised that I would only need to query the table cells on the webpage and see if they were marked as available somehow. Indeed I was in luck, each available table cell had the available CSS class assigned to it. That meant I could use HTML parsing libraries such as Nokogiri to analyse the page and find the relevant elements.

Next I checked the robots.txt file to see if there was any guidance to not crawl the site. In any case I figured that the traffic I would be generating would be minimal, as I would only have to check for availability once every hour or day or so. I didn’t find anything specific to dissuade me from my mission.

I then started writing a little ruby script, using the Watir to simulate the browser. Originally I tried just using other popular crawling gems such as plain Nokogiri or Mechanize but realised that they could not sufficiently deal with the AJAX calls that the site was using. Instead Watir uses a proper browser of your choice which you can pass a wait time to. At first I went with the Chrome web driver. Later when deploying this to Heroku, I switched to Phantomjs since Heroku doesn’t support apps running as a graphical based app, and the headless Phantomjs browser worked just as well.

The other bit to mention would be Mailgun. This is the first time I’ve dealt with sending emails from a service, and Mailgun was pleasantly simple to use, once I had found the right tutorials and realised you can send emails straight from the dummy domain that Mailgun gives you out of a box.

Finally I deployed everything to Heroku, and use the schedule plugin to set appropriate intervals to run the script. It would then email me whenever tickets became available for a target date of my choosing. I was pleased to find that I could run everything on free accounts, so didn’t actually have to spend any money to host anything! The script could obviously much more clever and configurable, but it was a fun project for a few hours, and most importantly I got my free tickets to Sky Garden!!

I’m aiming to put the code up on Github at some point, will post a link here! In the mean time, see the code below:

TicketChecker class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
  require 'watir'
  require 'pry'

  require_relative 'emailer'
  require_relative 'availability_parser'

  class TicketChecker
    TARGET_DATE = 4
    TARGET_MONTH = "June"
    BOOKING_URL = "https://skygarden_url_for_booking_tickets"
    BROWSER_WAIT_TIME = 5

    def call
      if checker.is_available?
        text = "TARGET DATE (#{TARGET_MONTH} #{TARGET_DATE}) IS AVAILABLE"
        emailer.send_email(text)
        puts text
      end

      browser.close
    end

  private

    def checker
      @_checker ||= AvailabilityParser.new(TARGET_DATE, TARGET_MONTH, browser)
    end

    def browser
      @_browser ||= begin
        web_browser = Watir::Browser.new :phantomjs
        web_browser.goto(BOOKING_URL)
        web_browser.driver.manage.timeouts.implicit_wait = BROWSER_WAIT_TIME
        web_browser
      end
    end

    def emailer
      @_emailer ||= Emailer.new
    end
  end

  TicketChecker.new.call

AvailabilityParser class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
  class AvailabilityParser
    AVAILABLE_KEYWORD = "available"

    attr_reader :target_day, :target_month, :browser

    def initialize(target_day, target_month, browser)
      @target_day = target_day
      @target_month = target_month
      @browser = browser
    end

    def is_available?
      available_tds_text.include?(target_day.to_s)
    end

  private

    def available_tds_text
      available_tds.map(&:text)
    end

    def available_tds
      tds.select { |c| c.class_name.include?(AVAILABLE_KEYWORD)}
    end

    def tds
      browser.iframes.last.tds
    end
  end

Emailer class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
  require 'mailgun' 

  class Emailer
    FROM_DOMAIN = "mail_gun_domain.com"
    TO = "myemail@gmail.com"
    SUBJECT = 'Skygarden Ticket Availability'
    API_KEY = "MAILGUN_API_KEY"

    def initialize
    end

    def send_email(text)
      client.send_message(FROM_DOMAIN, sender_params.merge(text: text))
    end

  private

    def client
      @_client ||= Mailgun::Client.new(API_KEY)
    end

    def sender_params
      { 
        from: "mailgun@#{FROM_DOMAIN}",
        to: TO, 
        subject: SUBJECT,
      }
    end
  end