/gempesaw/writing

About

2013-01-01T00:00:00Z

My name is Daniel Gempesaw, and I am a QA software developer for Sharecare. I create tools to improve our QA process, including an in-house browser automation suite based on Webdriver. I use Emacs for as many things as I can; recently I've been writing mostly Scala, Perl, elisp, and PHP in it. Anything Emacs, software, or web-dev related is on elisp-solves-problems: so far it's mainly explanations and snippets of my .emacs files.

I've also got a non-software tumblr at danzorx for other things that I enjoy like ultimate, snowboarding, and riding bicycles. I'm sometimes a pianist, bassist, and a novice ukulele-ist, but without a band or a gig in mind I'm getting a bit rusty.

I'm on Google+, and sparingly on twitter, or you can shoot me an email.

syntax highlighting this tumblr blog

2013-03-14T00:00:00Z

Syntax highlighting on tumblr is a completely manual effort, as far as I'm aware. I don't know of any built-in highlighting or any certified methods that they advertise. I did some research and tried a couple different highlighters - some of them are out of date, some of them don't support lisp, and some of them are more complicated. I was looking for a decently recent one that had built-in lisp support and would be easiest to hook into a tumblr set up. I settled on highlight.js, which has helpful documentation on its own website and everything.

Because I needed somewhere to host the files, I got set up with Github Pages and put my versions of the highlight.pack.js file in there. That took a while, but once it was sorted, the rest was pretty straight forward. I just need to select and add a CSS stylesheet with the colors of my choosing, add one script tag to source the js file that does the work, and then call javascript command to do all the highlighting. So, the following lines have been added to my tumblr theme:

<link rel="stylesheet" href="http://gempesaw.github.com/stylesheets/ir_black.css" />
<script type="text/javascript" src="http://gempesaw.github.com/javascripts/highlight.pack.js"></script>
<script type="text/javascript">hljs.initHighlightingOnLoad();</script>

Stylesheet, highlight.js, and activation. A fancy part of highlight.js is that it has language autodetection, so if it figures out I'm writing Lisp on its own, that's great! Sometimes, it misses though, and I have to manually wrap my source code in <pre><code class="lisp">source-code</code></pre> tags, which isn't that bad.

keeping my emacs windows balanced

2013-03-15T00:00:00Z

I've decided to get a bit OCD about my window sizes in Emacs, and I got tired of hitting C-x + to do balance-windows all the time. I'm not sure if there's a better way to do this, but I just added some advice around all of the commands I usually use to change the window layout: split-window-below, split-window-right, and delete-window.

(defadvice split-window-below (after restore-balanace-below activate)
  (balance-windows))

(defadvice split-window-right (after restore-balance-right activate)
  (balance-windows))

(defadvice delete-window (after restore-balance activate)
  (balance-windows))

So now my windows automatically balance themselves whenever I C-x 2 or C-x 3, all the time! Nifty. On the off chance that I'd like to manually control the size of a window, I've found enlarge-window under C-x ^; along with a C-u and C-x z, I can enlarge a window by a bunch of lines pretty quickly.

automatically save your bookmarks file

2013-03-16T00:00:00Z

Bookmarks in Emacs are very, very useful for getting to places you frequently go. If you don't use them to help you navigate, you should consider it! For whatever reason, bookmarks are saved in an external file, not in an Emacs variable. I can't count the number of times I added a bookmark, closed Emacs, and inadvertently deleted it. Even with version control, I still kept losing bookmarks because I kept forgetting to save.

(setq bookmarks-default-file "~/.emacs.d/bookmarks")
(defadvice bookmark-set (after save-bookmarks-automatically activate)
  (bookmark-save))

Let Emacs know where you want to save the bookmarks, and then use some advice! This little advice saves you all that woe: any time you set a bookmark, Emacs now saves all of them for you automatically. Presto :)

circe - an alternative to erc :)

2013-03-17T00:00:00Z

I recently heard about a new version of circe coming out so I went to try it out. I don't remember why I stopped using erc but afaik they both seem fine to use.

(package-install 'circe)
(setq circe-network-options
      `(("Freenode"
         :nick "dgempesaw"
         :channels ("#emacs" "#selenium")
         :nickserv-password ,freenode-password
         )))
(circe)

That should be enough to get you up and running and autoconnected to #emacs and #selenium. You'll have to setq your password in freenode-password, of course. Unfortunately, circe comes with tracking-mode which fights with ace-jump-mode for power over the C-c C-SPC keybinding. I added ace-jump-mode to that as well as its default at C-c SPC to help me whenever I'm clusmy. So, I'll just redefine the keys in the tracking-mode-map to my liking:

(eval-after-load "tracking"
  '(progn
     (define-key tracking-mode-map (kbd "C-c C-SPC") 'ace-jump-mode)
     (define-key tracking-mode-map (kbd "C-c C-@") 'ace-jump-mode)
     (define-key tracking-mode-map (kbd "C-x C-j C-k") 'tracking-next-buffer)))

C-x C-j C-k is close to jabber's C-x C-j C-l - all the new-message-buffer-cycling keys together, please! Otherwise, no complaints really, a pleasant time was had by all.

In fact, my first day idling in #emacs, I found out that you can bind the function keys sequentially like prefix keys, which OF COURSE makes sense but it opens up a whole new set of keybindings. For example,

(global-set-key (kbd "<f5> <f6>") 'do-something-cool)
(global-set-key (kbd "<f5> <f5>") 'do-someting-else-cool)

Is that not the coolest? I felt kind of silly for a bit but luckily Fuco hadn't realized it either, and people write about Fuco all the time so nyahhh!

clearing pngs off of my desktop

2013-03-18T00:00:00Z

A huge part of doing QA is taking screenshots and videos. Luckily, OS X makes it super easy to take screenshots with Command + Shift + 4. Unluckily, doing that all day long every day means the desktop gets super cluttered with your beautiful bug report pngs! Instead of manually deleting 'em, let Emacs do it for you :)

(defun delete-all-pngs-on-desktop ()
  "Opens a dired to desktop, marks all pngs, and tries to delete
them"
  (interactive)
  (save-window-excursion
    (dired "~/Desktop")
    (revert-buffer)
    (dired-mark-files-regexp "png" nil)
    (dired-do-delete)))

Go to the desktop folder in dired, refresh so it's up to date, mark all the pngs, and delete 'em. Time saved, pew pew!

fetchnotes and emacs, together at last

2013-03-19T00:00:00Z

Fetchnotes is yet another todo-list/note taking app. I like their service because everything is just plain text, so it's very straightforward from a usage standpoint. I don't have to worry about formatting or saving a url or just part of a page or anything that I'm not concerned about; it's just text! Additionally, they use tags to organize your notes; I have an affinity for tagging/labelling things excessively with the assumption that future me will be able to think of at least one of the tags past me used.

They've got a web interface and a nice responsive mobile view, and they also have an iOS app, an iPad app, and an Android app, which means that I can CRUD notes from any device I've got; since it's just text, it's easy on every device. As is the Emacs way, a thing worth doing is worth doing from inside Emacs. I've written fetchmacs, an Emacs major-mode to interact with their API from inside of Emacs. I've been using for myself for a few weeks.

It's pretty straightforward - the biggest issue that I always have with Elisp is figuring out how to dereference the data structures. Otherwise, it's just basic auth stuff and CRUD, with some keybindings to smooth things along. I haven't gotten around to MELPA/Marmalade integration, although that's my next goal. For the time being I've just been load-fileing it:

(defvar fetchmacs-user-email "user@n.ame")
(defvar fetchmacs-user-pass "password")
(load-file 'fetchmacs.el)

Put your user and pass in fetchmacs-user-email and fetchmacs-user-pass and you should be good to go. Invoke fetchmacs-view-notes to get to the notes dashboard, using fetchmacs-view-notes-mode. I've got it bound as C-x f, because I never use set-fill-column:

(global-unset-key (kbd "C-x f"))
(global-set-key (kbd "C-x f") 'fetchmacs-view-notes)

I really enjoy using magit, so whenever I possible I tried using the same or similar keys.

RET or e or o : fetchmacs-view-edit-note-at-point
c : fetchmacs-create-new-note
D or k : fetchmacs-delete-note-at-point
/ : fetchmacs-search
g : fetchmacs-refresh
n : fetchmacs-goto-next-note
p : fetchmacs-goto-previous-note
t : fetchmacs-filter-by-tag

Outside of fetchmacs-view-notes-mode there's also an editing mode which pops up in another window, just like editing magit commits. In there, the the following keybindings are available:

C-c C-c : fetchmacs-save-note
C-c C-k : fetchmacs-cancel-edit

So just like magit, you can save or cancel from in there.

And that's about it! On the off chance that a) there's actually anyone reading this, b) you're an Emacs user, c) and you're a Fetchnotes user - a pretty unlikely cross section - please give it a try! :)

convenience functions: eol punctuation and creating new lines

2013-03-20T00:00:00Z

Writing perl and php, I often find myself needing commas or semicolons at the end of a line. After I got tired of doing C-e ; <return> and C-e , <return> all the time I decided I'd make things easier on myself.

(global-set-key (kbd "C-,") (lambda () (interactive)
                              (end-of-line)
                              (insert ",")
                              (indent-according-to-mode)
                              (forward-line 1)
                              (indent-according-to-mode)))

(global-set-key (kbd "C-;") (lambda () (interactive)
                              (end-of-line)
                              (insert ";")
                              (indent-according-to-mode)
                              (forward-line 1)
                              (indent-according-to-mode)))

I bound C-, and C-; to get to the end of the line, insert the appropriate character, indent, go down a single line, and indent the new line, too. I find them to be useful because it allows me to finish up a line of code from anywhere on the line. I also did something similar on C-<return>:

(global-set-key (kbd "C-<return>") (lambda () (interactive)
                                     (end-of-line)
                                     (reindent-then-newline-and-indent)))

Instead of inserting any punctuation, this just opens up a new line beneath the current one with the proper identation and moves the cursor there. These are pretty small little things but I use them multiple times a day to save me numerous keystrokes.

setting up jabber.el

2013-03-21T00:00:00Z

I recently added jabber.el to my workflow via package-install jabber. Jabber.el seems to be in a really good place in terms of usage and polished development. The only hiccup I had was that it was complaining about starttls issues. That was easy enough to google, and it seemed like I should specify the connection type as starttls. We're apparently using an expired cert or something, so I had to make starttls insecure as well. I got the impression I should be more worried about that, but oh well.

(setq jabber-account-list '((work-email-address
                             (:connection-type . starttls))))
(setq starttls-extra-arguments '("--insecure"))
(setq starttls-use-gnutls t)

So that got me connected. As always, there were some customizations that were necessary. First, I don't care about seeing presence, and I don't care about who is offline:

(setq jabber-alert-presence-hooks nil
      jabber-show-offline-contacts nil)

Next, I don't want to hear about the avatars, I don't even want to download them, and I don't want them showing up in the roster:

(setq jabber-avatar-verbose nil
      jabber-vcard-avatars-retrieve nil
      jabber-roster-line-format " %c %-25n %u %-8s (%r)")

I do want the history, as well as alerts in my mode line, and we can shorten the buffer names for funsies.

(setq jabber-history-enabled t
      jabber-mode-line-mode t
      jabber-chat-buffer-format "*-jabber-%n-*"
      jabber-roster-buffer "*-jabber-*")

Finally, I want to autojoin some chatrooms every time I connect:

(setq jabber-muc-autojoin '("qa@conference.sharecare.com"))

All together, I'm quite happy with my jabber setup. Pretty low amount of customization necessary for being able to do away with Adium entirely.

Oh, and keybindings - I only know and use a few of them

C-x C-j C-c - connect! start out with this one obviously
C-x C-j C-l - if you have a pending message, switch to that buffer! if you've already switched to a pending message buffer, switch back to the one you came from if there are no more new messages! if there's no pending messages, do nothing! I really like this one.

My only complaint is that I wish I started using jabber.el earlier, as it's quite nice to stay in Emacs for chats; quite nice, indeed.

save-window-excursion: protect your window layout

2013-03-22T00:00:00Z

Sometimes I've got my window layout set up just right, and then a command like tex-file stomps all over my hard work. winner-mode helps with one off cases, but tex-file always pops up a new window for the buffer containing the tex compilation. I usually don't care about the compilation, especially since it usually doesn't fail, so let's stop popping up that window:

(add-hook 'latex-mode-hook
          (lambda ()
            (define-key latex-mode-map (kbd "C-c C-f")
              (lambda ()
                (interactive)
                (save-buffer)
                (save-window-excursion
                  (tex-file))))))

tex-file is usually at C-c C-f, so let's replace it in the key map by my interactive function. It saves the buffer and uses save-window-excursion to keep everything in the same place. Hooray :)

markdown-preview with syntax highlighting

2013-04-01T00:00:00Z

In a previous post, I talked about adding syntax-highlighting to this blog. It's working out quite well for me, but I wanted to pull it into my composition workflow better. My current suboptimal workflow is to compose markdown in Emacs and copy/paste into Tumblr's web app. (Tumblesocks is having connection problems that I don't feel like figuring out quite yet.) When I markdown-preview, I want to be able to see my code in all its syntax highlighted glory. Let's look at markdown-preview:

(defun markdown-preview (&optional output-buffer-name)
  "Run \`markdown' on the current buffer and preview the output in a browser."
  (interactive)
  (browse-url-of-buffer (markdown markdown-output-buffer-name)))

So, looking into that fourth line, we call markdown with the argument markdown-output-buffer-name. markdown itself runs markdown on the current buffer and outputs to the optional argument: in this case markdown-output-buffer-name. Then, we pass that temporary buffer to browse-url-of-buffer which opens it up in Chrome and voila! we see a preview of the markdown under composition.

What I wanted to do was to add the same style and script sources that I added to the tumblr theme. That way, my markdown preview would have the same syntax highlighting that my tumblr would have! So, I wrote my own markdown-preview function:

(defun markdown-preview-with-syntax-highlighting (&optional output-buffer-name)
  "Run `markdown' on the current buffer and preview the output in a browser."
  (interactive)
  (browse-url-of-buffer
   (with-current-buffer (markdown markdown-output-buffer-name)
     (goto-char (point-min))
     (if (> (length markdown-css-path) 0)
         (insert "<link rel=\"stylesheet\" type=\"text/css\" media=\"all\" href=\""
                 markdown-css-path
                 "\"  />\n"))

     (if (> (length markdown-script-path) 0)
         (progn
           (insert "<script type=\"text/javascript\" src=\""
                 markdown-script-path
                 "\"  /></script>\n")
           (insert "<script type=\"text/javascript\">hljs.initHighlightingOnLoad();</script>")))
     markdown-output-buffer-name)))

Where I want to jump in is after markdown has processed my file but before we send the output buffer to browse-url-of-buffer. (markdown markdown-output-buffer-name) returns the output buffer, so I catch that into the first argument of with-current-buffer. I go to the beginning of the buffer and determine whether or not I know where the desired css and scripts are. Assuming they're defined somewhere else...

(setq markdown-css-path "/opt/highlight.js/src/styles/ir_black.css")
(setq markdown-script-path "/opt/highlight.js/build/highlight.pack.js")

the function will insert analogous style and script tags that highlight the text!

Last but not least, instead of calling markdown-preview as my compile command for markdown buffers, I should call my improved version:

(add-to-list 'smart-compile-alist '("\\.md\\'" . (markdown-preview-with-syntax-highlighting)))

Local automatic syntax highlighting in markdown previews complete!

Telling auth-source (and smtpmail-send-it) about my changed email password

2013-06-06T00:00:00Z

I recently had to change my email password, as properly dictated by work's IT policies. I use offlineimap to get my mail, mu4e to read it, and smtpmail-send-it to send it.

(setq message-send-mail-function 'message-send-mail-with-sendmail
      message-send-mail-function 'smtpmail-send-it
      smtpmail-stream-type 'starttls
      smtpmail-default-smtp-server "pod51019.outlook.com"
      smtpmail-smtp-server "pod51019.outlook.com"
      smtpmail-smtp-user "dgempesaw@sharecare.com"
      smtpmail-smtp-service 587)

Offlineimap manages its password stuff from a config file at ~/.offlineimaprc. I updated that file and I was able to continue retrieving my mail with no hiccoughs. I then updated my ~/.authinfo file as well with the new password, but I continued to get the authentication failed error.

Sending failed: 535 5.7.3 Authentication unsuccessful in response to base64-encoded-password

That was pretty odd, considering that I changed the source file and all. I tried digging through smtpmail.el a little bit, looking for occurences of password or auth or something. I happened to see a defun called auth-source-search which got me into auth-source.el. Doing an apropos-command for "auth" led me to auth-source-forget-all-cached.

Apparently the auth sources like ~/.authinfo are cached for a customizable length of time, so I just had to clear the cache and everything is back in running order!

On the E3 rape joke

2013-06-12T00:00:00Z

There was a rape comment during Microsoft's E3 demo. Many people are arguing that it wasn't a rape comment, mostly consisting of "I didn't parse it as a rape joke because (it's a stretch|I can think of something else it might refer to|I'm not oversensitive|.*), so it's not and stop complaining about it." It's narcissistic, misogynistic, and ignorant to assert to other people (!!!! who have entirely different experiences!) that it's equivocally not a rape joke and to try to stop conversation about it.

Regardless of anyone's dissenting opinion, the fact is that many people interpeted the comment as a reference to rape. This fact along with the context (Microsoft presenting to a huge audience practically guaranteed to include rapists and rape survivors in an industry plagued by gender issues) should be enough to make the entire situation reprehensible - why should it matter that there are some people who aren't offended, and why do they need to inject themselves into the conversation? The entitled attitude of "I'm not offended so you shouldn't be either" is infuriating.

Still, some people don't care about offending others; they can't be bothered with being polite, politically correct, tactful, minimally empathetic, or whatever they want to derisively call it. Even for these people, this is still a problem that we shouldn't shout down or drown out, because words matter, because rape jokes make rapists feel comfortable; because we shouldn't make rapists feel better.

On dudebros and sexism in games

2013-06-15T00:00:00Z

I despise it when people take a small sampling of truly sexist games (which I will readily admit exist, and some are legitimately good games despite the fact), but the same people choose to ignore the huge number of games with amazing, strong female characters.

It's not enough that games with strong female characters exist, or even that there's an (arguably) huge number of them. As I understand it, it's the pervasive sexism in the entire industry, manifested in things like: the actual game characters themselves, like when Microsoft didn't release a single game with a female lead at E3 this year; the attitudes executives of industry-leading game studios, like when they publicly (and incorrectly) opine that games with female protags don't sell; the bullshit the few female devs have to put up with, as seen when the #1ReasonWhy hashtag exploded, etc.

I'm tired of hearing uneducated hits at my favorite hobby. True gamers, not the common Halo or Call of Duty "dudebros" that make up the public front of gamers, are one of the least sexist groups one could possibly meet. Many of us, from very young ages, have been exposed to great female role models and characters.

The "true gamers" label is disingenuous and doesn't excuse anyone on either side of it of their behavior. Anyone who plays games is a gamer, and it's just as ridiculous to say that all Halo/CoD players are sexist as it is to say that "true gamers" are "one of the least sexist groups."

I'm going to mostly skip the entire ME section - the trilogy is probably one of the best franchises in terms of female representation, but it's still fraught with problems, like feministpoints mentions. I'm not quite sure what the list of female characters is supposed to do for the argument.

Alyx Vance from Half-Life 2, Half-Life 2: Episode 1 and Half-Life 2: Episode 2 is widely considered to be the greatest companion in any game, ever. Alyx is not sexualized in the story and is always portrayed as a great friend to the protagonist, Gordon Freeman. Faith from Mirror's Edge could have easily been taken advantage of with her loose clothing, but the developers chose not to, to respect her gender and character.

Using Alyx Vance as an example seems to be a double edged sword - sure, her character is a great companion, but who wants to grow up to be a great companion? People don't want to be Midna, or Robin, or Watson: they want to be Link, Batman, and Sherlock. Faith is, as far as I can tell without having played the game, a great example, but again, the existence of some or even many good games with strong female protags and supporting characters has isn't enough to counter the rest of the industry's rampant sexism.

Almost every game with character customisation allow males and females to be made by the player. I, as a male gamer, often choose to play female over males in these places, and so do many of my friends. Females are often more convincing as leaders and heroes in gaming, and the voice actors often give better performances and carry more charisma. Plus, it is always nice for at least one form of media to escape from normal sexist tropes surrounding the female gender.

Cool! These are nice anecdotes and opinions, but I don't think they have much bearing on the state of gaming.

Yes, there are many sexist and/or misogynistic games - Call of Duty, Gears of War, and Halo come to mind - but they are poor representations of the medium as a whole. Gaming is actually very progressive in the image of strong, independent, competent female characters, whether in side roles or as heroines. This wonderful fledgling medium is getting a far worse reputation than it deserves.

Yes, agreed! Some games have been very progressive with amazing heroines, and that is awesome. Gaming is an amazing and exciting medium with so much untapped potential, but to claim it's not sexist just doesn't hold up.

All that being said, there's nothing wrong with liking and loving these games and the industry in spite their sexism - it's okay for us to like problematic things. It doesn't make us a bad people, as long as we are cognizant of the shortcomings of the game and the industry. The ME series is one of my all time favorites, but it's far from perfect in many ways and that's okay! It's paving the way for better games in the future, and that's what is really exciting.

Lesser known Emacs helps

2013-06-17T00:00:00Z

I recently came across a pair of lesser known (for me) help keybindings: C-h c and C-h w. I ran through the tutorial on a flight because I haven't yet and I've been meaning to, and that's where I found C-h c which invokes describe-key-briefly. It's basically a baby version of C-h k, describe-key. Instead of popping up another window to tell you about the keystroke, it just puts a short little description in the minibuffer. I think it'll be pretty useful if I can remember to use it to keep from messing up my own window config.

Kind of going the other direction, C-h w invokes where-is which prompts you for an interactive function and tells you what, if any, keybinding it's attached to. This is pretty useful when I know the name of the command I want to do, but I'm not sure what the keybinding is for it. I ran across that situation with a couple magit commands the other day and I was able to work backwards to figure out the keybindings. Previously, I'd been doing C-h m or C-h b to get a list of keybindings and then C-sing through that list for the name of the command.

Of course, both of these (and more) are immediately available through C-h C-h, and they've been there for the entire time that I've been an Emacs user.

Using a command line argument for PhantomJS and the Selenium Server Standalone

2013-08-19T00:00:00Z

tl; dr - -Dphantomjs.binary.path=/path/to/your/phantomjs

Selenium WebDriver is a browser automation framework that uses "drivers" to drive browsers. So, there is a "chromedriver" that you send instructions like "click on the Sign In link" or "tell me what color that element is," and it will carry out the instructions in an actual Chrome browser. PhantomJS, by Ariya Hidayat, is a quick, lightweight headless browser that parses JS (unlike HTMLUnit! !). Since it doesn't have to render anything on the screen, it can perform these tests much faster, but the tradeoff is accuracy - you're no longer testing in the exact same environment as your users.

Its driver is cleverly (?) named GhostDriver, written by Ivan De Marino. GhostDriver is built-in to PhantomJS, so if you want to use PhantomJS in your WebDriver tests, the only thing you need to do is install PhantomJS, which Ariya makes very easy (for example, brew install phantomjs).

I'm using the Perl WebDriver bindings, and I start the selenium server with a shell command that looks something like

C:\> java -jar selenium-server-standalone-2.35.0.jar ^
    -Dwebdriver.chrome.driver=C:\chromedriver.exe ^
    -Dwebdriver.ie.driver=C:\InternetExplorerDriver.exe

That -Dwebdriver.chrome.driver option tells the Selenium server where to go find my Chromedriver, and likewise for IE. Now, PhantomJS makes it really easy to connect to an existing standalone server - from their Github page, once PhantomJS is in your path, you just need to

$ phantomjs --webdriver=<PORT_TO_WEBDRIVER> # default is 4444

So, those two commands will start up the Selenium standalone server and hook up PhantomJS to it, so you can request a PhantomJS browser of the server and run your tests. But, it's a pain to have to set up two processes for PhantomJS when I can already add the Chrome and IE drivers as options. I couldn't find this documented anywhere, but you can also add the PhantomJS binary as a command line argument to the standalone server:

$ java -jar selenium-server-standalone-2.35.0.jar \
    -Dphantomjs.binary.path=/path/to/your/phantomjs

If you use homebrew, it would be

$ java -jar selenium-server-standalone-2.35.0.jar \
    -Dphantomjs.binary.path=/usr/local/bin/phantomjs

Unknown command 'WaitForAllTabsToStopLoading'

2013-08-21T00:00:00Z

I woke up this morning and went to run some Webdriver jobs in Chrome, but for some reason they weren't working! The browser would start up, and then it would die, complaining about an unknown command called 'WaitForAllTabsToStopLoading'

Dying: Error while executing command: An unknown server-side error occurred 
while processing the command.: Unknown command \'WaitForAllTabsToStopLoading\'.

It seems like there's a problem with older versions of Chromedriver and the newest version of Chrome. While my day to day Chrome hasn't updated since I never close it, the Chrome that Webdriver uses gets restarted every time a test is run, so it updated itself to Version 29.0.1547.57. As a result, my old Chromedriver doesn't seem to know how to drive the new Chrome. Updating the chromedriver binary to the one released on August 6 fixes the problem for me.

For a few minutes, this was a really confusing bug. I'm constantly tweaking our automation framework, so I wasn't too surprised when the tests were failing - I figured I must've broken something on my dev branch of the framework. However, when a couple of my coworkers said they had the exact same issue, despite not changing anything from the day before, I had no idea what might be going wrong. I didn't think about Chrome updating itself on all of our computers at the same time, leading to duplicate bugs in systems that were supposedly unchanged. Fun :)

I don't know anything about SQL Server 2008 R2

2013-09-18T00:00:00Z

One of my work assignments depended on an install of SQL Server 2008 R2. I needed to use the sa account to log in and do some database-y things during latter parts of the install, but I didn't know the password for the sa account, and I didn't know how to find it or change it. According to Google, SQL Server Management Studio is what I need to do admin things. At this point, I found out that the sa account was actually disabled by default in new installations of 08 R2 as a security precaution, so my task was to get admin access, enable the sa account, and change its password.

There are apparently two methods of login for SQL Server 2008: Windows Authentication mode, wherein you use the same credentials that you do for logging in to the machine itself, and SQL Server and Windows Authentication mode. The latter allows us to use SQL logins alongside of the Windows credentials. Now, enabling the sa account wouldn't have done anything on its own if the server authentication mode was Windows Authentication only, which would have prevented me from logging into the sa account. So.

First, in order to do admin things to other accounts, I apparently needed to temporarily enable single user mode for the server, described in a Microsoft article I had been lucky enough to find off stackoverflow. After changing the startup command and restarting the server, I was able to use SSMS with no restrictions and full freedom to muck about with all sorts of stuff.

Next, I had to change the server authentication to "Mixed" so that enabling the sa account was worth anything at all. Docs for that courtesy of another Microsoft article.

Finally, I had to open the properties of the "sa" acconut up, "Enable" it, and change its password to something I knew.

Hmm, I've learned that I don't like to use GUIs to manage my databases, and that VM snapshots are quite useful after you've done some crazy things to the db. Useful, I guess, but trouble shooting slow GUIs inside of a slow VM was super distracting, causing me painful context switches all day long.

js2-mode timer errors: timer-set-function: Wrong type argument: timerp

2013-09-25T00:00:00Z

I started seeing the following timer error in mooz's js2-mode, the actively maintained fork of Steve Yegge's js2-mode. This would happen whenever I changed any javascript code, and the mode wouldn't rehighlight the new code. Invoking js2-mode interactively would properly apply the syntax highlighting, but then the next change to the file would cause the error again.

 timer-set-function: Wrong type argument: timerp, [t nil nil nil nil nil nil nil]

There is also a fork of this fork, js3-mode, which was working fine for me, but unfortunately skewer-mode is only compatible with the AST generated from js2-mode. I'd guess that making skewer-mode compatible with js3-mode might be just a change to a variable name or something, but I wasn't too interested in digging into that on my own.

I'm unfortunately not sure what caused the issue for me. According to the solution, it seems to have been updating to Emacs 24.3 instead of 24.2, but I hadn't recently updated my Emacs. Either way, I was fortunate enough to find a solution in an issue on Github. It seems that the culprit is stale bytecode compiled by 24.2 that is invalid on 24.3 So, as described in the issue, recompiling js2-mode and restarting Emacs fixed it for me as well:

(byte-recompile-directory
  (expand-file-name "~/.emacs.d/elpa/js2-mode-20130307.2012/") 0 t)

You'll have to replace the js2-mode version string with the date of your actual js2-mode folder. byte-recompile-directory also works recursively, so it can be applied to the ~/.emacs.d/elpa folder to recompile all of the elpa packages, or the ~/.emacs.d/ folder to do your entire init folder.

How do you test a big ReST API binding?

2014-04-03T00:00:00Z

I recently became the maintainer of Selenium::Remote::Driver, which is the Perl bindings to the Selenium Webdriver API. I just released version 0.18 to the CPAN and hopefully got the tests passing again :D.

I'm still trying to figure out the Best Way to test a module that is basically an interface to a ReST API. As far as I'm aware, integration testing these kinds of modules basically require you to mock or fake out network responses, but there were a couple articles on blogs.perl.org that discouraged the use of mocks, so I'm a bit unsure about best practices.

Test::LWP::UserAgent is one way to mock/fake out network traffic in Perl, and it's similar to AngularJS's $httpBackend: you instantiate a fake network object, and then tell it what requests it should expect. You inject the fake network object into your module's constructor, and when it tries to make a request, the mock catches it and lets you examine the request for tests and the like. You can also specify responses to test out the different branches of the module, like for error handling tests.

For a big module, having to set up tests for every single request can get a bit cumbersome, so S::R::D went the way of simply recording all of the requests/responses that its test suite makes, and including the recordings with the module. When running the tests, it loads up the recording and matches incoming requests to its list of recorded REQ/RES pairs, and returns the appropriate response.

Unfortunately, the recording process is different across platforms, so every change to the module or the tests require new recordings for each supported platform. Additionally complicating the issue was that there was a bug somewhere in the recording code, so it would only work with an LWP-Protocol-PSGI-0.04, which is the module used to record the requests and responses.

bookmarking magit buffers with projectile

2014-04-04T00:00:00Z

For the longest time, I've wanted to be able to bookmark the magit buffers of my various git projects. It's really useful to be able to jump to the top level view and get a git status summary of what I was in the middle of. However, I wasn't able to use Emacs's built in bookmarks for this, because those require a file reference to jump to, and the magit buffers were not file based. Enter Projectile :)

Projectile (Project Interaction Library for Emacs) by Bozhidar Batsov is a really nifty tool for managing different projects in Emacs. For me it primarily smooths out finding files in projects, and switching between project directories. In particular, I'v got s-p bound to projectile-switch-project:

(define-key projectile-mode-map [?\s-p] 'projectile-switch-project)

Projectile keeps a cache of the different git directories I've been to, and provides it as a list of choices. You can customize the action that Projectile does after you select a project, and there are a few suggestions like projectile-find-file, which would prompt you for a file to visit, and projectile-dired, which immediately opens up the new project's directory in a dired buffer. Both useful, and for a while I settled for opening the project in a directory and invoking magit-status myself.

However, my new favorite is

(setq projectile-switch-project-action 'projectile-vc)

which tells projectile to open up the vc buffer for the new project, which would be the magit buffer for git projects. Itch scratched!

convenience fns: open-jira-ticket-at-point

2014-04-10T00:00:00Z

Working with JIRA, there are a bunch of instances where I see a JIRA ticket inside of Emacs and I want to open it in Chrome - they come up often in jabber chatrooms, there's tons of them in emails, etc. No reason for me to copy it manually or search for it again on JIRA - just let Emacs do the work. :D

(defun open-jira-ticket-at-point ()
  (interactive)
  (let ((ticket (thing-at-point 'sexp)))
    (if (eq nil (string-match-p "^[A-z]+-[0-9]+$" ticket))
        (setq ticket (read-from-minibuffer
                      "Not sure if this is a ticket: "
                      ticket)))
    (browse-url
     (concat
      "http://arnoldmedia.jira.com/browse/" ticket))))

(thing-at-point THING-TYPE) is a useful function that "returns the THING at point". It accepts a couple different options of things that it tries to look for - in our case, sexp seems to be reasonable at picking out JIRA tickets from different contexts. Sometimes, it picks up some extra stuff and we need to manually fix the input. So, if the thing-at-point doesn't match the regexp "^[A-z]+-[0-9]+$", then ask the user to fix it a little bit, and then go ahead and browse-url the ticket in Chrome. Easy peasy.

The better thing to do is probably go into thingatpt.el and figure out how to make my own thing that it recognizes, but I couldn't be bothered at the time.

travis build badges, POD, dist::zilla and pod::weaver

2014-04-12T00:00:00Z

I've taken the plunge with the excellent Dist::Zilla recently for authoring my Perl distributions. Since picking up the maintainer role of S::R::D, D::Z fit my needs perfectly: speed up distribution and alleviate all the repetitive parts of the process.

I recently started work on Perl bindings to the Browsermob Proxy REST API called, unimaginatively, Browsermob::Proxy. As usual, we want it to run its tests on travis, and there's the lovely [TravisYML] plugin for that:

# dist.ini
[TravisYML]

[GatherDir]
include_dotfiles = 1

[PruneCruft]
except = \.travis.yml

[PodWeaver]

[ReadmeAnyFromPod]
type = markdown
filename = README.md
location = root

and we also want to use Pod::Weaver for more of that DRY goodness. Since we're writing all the documentation in the main module anyway, [ReadmeAnyFromPod] generates a markdown readme for us that Github likes. The last bit is getting the Travis build status badge into that readme. Surprisingly, Travis even have a suggested form for POD badges:

=for HTML <a href="https://travis-ci.org/gempesaw/Browsermob...

But unfortunately that won't work in our case, since we're outputting to markdown. Via trial and error, I found that using the markdown form of the build badge in a =for markdown POD section works with [ReadmeAnyFromPod] with type = markdown:

=for markdown [![Build Status](https://travis-ci.org/gempesaw...

in the POD does get it into the generated readme, but since it's not in a dedicated section, [Pod::Weaver] just lumped it into the [Leftovers] section, which by default is near the end of the document. So, I moved leftovers up to the top and we're in business!

# weaver.ini
[@CorePrep]

[-SingleEncoding]

[Name]
[Leftovers]
[Version]
...

(This may be troublesome if other things fall into the Leftovers section, so we'll have to keep an eye out...)

best practices for angular project structure

2014-04-17T00:00:00Z

A few months ago, the AngularJS team published a document outlining a new best-practices project structure. There's also a (huge) Github issue discussing its pros and cons in finer detail, and it was mentioned in the generator-angular 2014 Q1 roadmap issue.

The old recommended structure, which is what yo angular generates, groups files together by function: controllers are with other controllers, directives are with other directives, etc. In some cases, they're all even in the same file.

app/
    css/
        app.css
        bootstrap.css
    scripts/
        app.js
        controllers/
            main.js
            sidemenu.js
        services/
            backend.js
        directives/
            datepicker.js
    templates/
        datepicker.html
        main.html
    index.html
test/
    spec/
       controllers/
           mainSpec.js
       services/
           backendSpec.js
       directives/
           datepickerSpec.js

That was easy for me to grasp initially and due to the strong tooling built around that structure, it was easy to add new directives and providers because the tools knew how to navigate the folder structure and set up the boilerplate. But, I really like the new proposed structure due to some shortcomings of the old one:

It doesn't scale very well. In a large project, the controllers, directives, and services folder will each have many files, and the majority of them will be unrelated to each other.
It's hard to find things if you don't know where to look, or if you've just forgotten where you put it. Given a front end component that in which you found a bug, it's not immediately evident where you would look to find and edit that functionality.
When editing a file, it's not immediately evident from the filename what kind of file you're looking at, as the 'type-of-thing' classification is done in the folder name.

The new structure is grouped by components and might look like

app/
    app.js
    index.html
    components/
        datepicker/
            datepicker-directive.js
            datepicker-directive_test.js
            datepicker.html
        backend/
            backend-service.js
            backend-service_test.js
    main/
        main-controller.js
        main-controller_test.js
        main.html

The components are grouped by functionality and house services, filters, directives, and related files (templates, test files, etc). Meanwhile, the sub sections like main would only have controllers, css, and templates. I really enjoy using the new layout and it makes it super easy to switch between the test and the implementation, since they're right next to each other.

When I converted my project to this structure, I had to account for the new file structure in the Gruntfile and in the karma configuration. Both of the config files have a useful **/*.js glob option that recursively includes all child files that match, but the bower_components folder also gets picked up, and then I end up linting someone else's files.

My current project only has one sub-section, editor, so the end of the karma.conf.js files section looks like:

 // our scripts
 'app/app.js',
 'app/editor/**/*.js',
 'app/components/**/*.js'

The gruntfile was a bit more of a pain, since the js files are referenced in more places. But, the watch:livereload task after the conversion looked like:

livereload: {
    options: {
        livereload: '<%= connect.options.livereload %>'
    },
    files: [
        '<%= yeoman.app %>/{components,editor}/**/*.{js,html,css}',
        '<%= yeoman.app %>/index.html',
    ]
}

It's a bit unwieldy - the a/{b,c}/e construct matches a/b/e and a/c/e, and the **/* does a recursive descent into folders, so that one line catches everything in components and in my only subsection, and skips the bower_components folder.

An unexpected dump_config error in Dist::Zilla projects using [TravisYML]

2014-04-20T00:00:00Z

1. motivation

There's been a curious dump_config build error in my Dist::Zilla Travis-CI builds as of late. I'm using the [TravisYML] Dist::Zilla plugin to generate my .travis.yml file, and up until now it's been working great. But, there seems to be some dependency conflict that I don't have on my local machine - at some point, the travis build box can no longer run any dzil commands, as it bombs out with:

The method 'dump_config' was not found in the inheritance hierarchy
for Dist::Zilla::App::CommandHelper::ChainSmoking at [...]

Which is then followed by a completely unhelpful (to me) Moose stacktrace.

2. debugging travis builds?

I wasn't aware of a good method to debug travis builds - previously, I had been just committing minor changes, pushing 'em to Github, and then waiting for the Travis build to kick off. That feedback loop could take up to 15 minutes, and wasn't really conducive to any sort of debugging method.

I emailed support AT travis-ci.com inquiring about getting access to one of their build boxes, since the outdated vagrant box method was now obsolete and one of their support staff got back to me within minutes with a cloud box I could SSH into for 24 hours. Really pleased with their turnaround time and support, considering it's a free service!

3. perlbrew on travis debug boxes

The box I got had a ~/perl5 folder with a bunch of perls already installed. But, it didn't have the perlbrew binary in the path. I updated the path

$ export PATH=~/perl5/perlbrew/bin:$PATH

but the commands like perlbrew use and perlbrew switch weren't having any impact on perl -v. It turns out I needed to

$ perlbrew init
$ source ~/perl5/perlbrew/etc/bashrc

in order to get perl -v to recognize I didn't want to use the system perl anymore. Had I been using more shells, I'd probably want to put all of that in the ~/.bash_profile, but I only needed the one, especially since perlbrew lib lets you manage completely separate local::libs.

So, I switched to 5.19 and spun up a new local::lib and started playing around with dependencies. If I got too far and broke everything again, a new local::lib can reset all the installed deps and get me into pristine condition.

$ perlbrew switch 5.19
$ perlbrew lib create debug\_dist\_zilla
$ perlbrew use 5.19@debug\_dist\_zilla

4. getting to the heart of the issue

The generated .travis.yml file does a couple of things - it exports some global variables, configures a bit of necessary git settings, and then gets to the business of handling author and project dependencies.

$ cpanm --quiet --notest --skip-satisfied Dist::Zilla
$ dzil authordeps | grep -vP '[^\\w:]' | \
    xargs -n 5 -P 10 cpanm --quiet --notest --skip-satisfied"
$ dzil listdeps   | grep -vP '[^\\w:]' | cpanm --verbose --skip-satisfied"

The travis build boxes come with 5.006 installed, so the first line actually doesn't do anything. As reported in the github issue, the dump_config bug occurs at the dzil listdeps step. So, one of the authordeps must be causing it ... probably.

$ dzil authordeps --missing | head -n 1 | cpanm --skip-satisfied && dzil

It just steps through each of the author dependencies one at a time, installing it with cpanm, and then verifying that I can still dzil afterwards.

5. solutions?

In not-so stunning turn of events that seems incredibly obvious now, the problem seems to arise after installing Dist::Zilla::Plugin::TravisYML. In the generated travis script, the author deps are installed with cpanm --notest, so it's not caught until the next time we attempt to dzil listdeps. Stepping through the tests one by one, the ::TravisCI install actually fails its own tests with the same dump_config error that we see in the travis logs after --notest installing it.

So, in theory, commenting out the [TravisYML] from dist.ini should remove it from dzil authordeps and we should be good to go. I'm going to go try that now...

Getting started with Ionic and Appium for a new android app

2014-04-25T00:00:00Z

Ionic enables you to author mobile apps using the powerful AngularJS javascript framework. Appium is a cross-platform mobile testing tool that lets you run your app on emulators and real devices. Both of these open source projects have full support for iOS and Android, which is just great. This morning, I worked through taking a generated Ionic app all the way to my device for testing with Appium.

[[MORE]]

0. prereqs

You'll need ionic (which depends heavily on cordova), and appium. Ionic needs ant to build the android app; if you're using Homebrew on OS X, they've gotcha covered.

$ npm install -g cordova ionic appium
$ brew install ant

You'll also need the Android SDK, which has the Android Developer Tools in it, I think. I don't know of a better way to install them other than downloading the zip and unzipping it to a known folder. After unzipping the sdk somewhere, you'll want to add sdk/tools and sdk/platform-tools to your path, and you'll also want to export ANDROID_HOME. Here's the relevant section of my ~/.bash_profile; you'll need to edit ANDROID_HOME accordingly.

export ANDROID_HOME=/opt/adt-bundle-mac-x86_64-20140321/sdk
ANDROID_TOOLS=$ANDROID_HOME/tools/:$ANDROID_HOME/platform-tools
export PATH=$ANDROID_TOOLS:$PATH

Appium will also want you to export $JAVA_HOME; stackoverflow says you can do it like this on OS X Mavericks:

export JAVA_HOME=$(/usr/libexec/java_home)

1. the ionic half

Seeing as I'm not too concerned with the app itself right now, just working on getting all the pieces in place, we'll just use generator-ionic as our template.

$ npm install -g generator-ionic
$ yo ionic

That'll lead you through a few set up steps and then generate a boilerplate app that should be enough to make a baby app. Next thing to do is to specify that you want android as a platform target:

$ ionic platform add android

At this point, you'll need to have ant and android installed and in your $PATH, or else the command will fail; see the prereqs part above if you're missing anything. Pay a little bit of attention to the output of this command - in particular, you'll want to hold on to the Package and Activity lines for firing up Appium via desired capabilities. I called my toy app Breakside, so the relevant lines for me said:

Creating Cordova project for the Android platform:
    Path: platforms/android
    Package: com.example.Breakside
[...snip...]
No project name specified, using Activity name 'Breakside'.

After that, you just have to build the .apk for appium to use:

$ ionic build android

That should create a couple .apks in platforms/android/ant-build, including one called suffixed -debug.apk. That's what I used for with Appium in the next step.

2. the appium half

I ran Appium with my Galaxy Nexus hooked up to my laptop via USB, with the USB debugging mode on; you'll need to do the same if you want to run on a device^[1]

Appium comes with a useful binary called appium-doctor which should be in your path if you've installed it with npm. You can run it to double check whether your system is ready for testing Android devices.

$ appium-doctor
Running iOS Checks
[...snip...]

Running Android Checks
✔ ANDROID_HOME is set to "/opt/adt-bundle-mac-x86_64-20140321/sdk"
✔ JAVA_HOME is set to "/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home."
✔ ADB exists at /opt/adt-bundle-mac-x86_64-20140321/sdk/platform-tools/adb
✔ Android exists at /opt/adt-bundle-mac-x86_64-20140321/sdk/tools/android
✔ Emulator exists at /opt/adt-bundle-mac-x86_64-20140321/sdk/tools/emulator
✔ Android Checks were successful.

✔ All Checks were successful

If you already exported $ANDROID_HOME and $JAVA_HOME, everything should be good to go! Run the appium command after making sure that the shell knows about $ANDROID_HOME and $JAVA_HOME:

$ appium &

The next thing is figuring out how to tell Appium what you want to do. I'm using the Perl Webdriver bindings, but luckily desired capabilities are the same in any language. Here's my short perl script script:

#! /usr/bin/perl

use strict;
use warnings;
use Selenium::Remote::Driver 0.1951;

my $caps = {
    'app' => '/opt/breakside/platforms/android/ant-build/Breakside-debug.apk',
    'app-package' => 'com.example.Breakside',
    'app-activity' => 'Breakside',
    'browserName' => '',
    'platformName' => 'android',
};

my $android = Selenium::Remote::Driver->new_from_caps(
    port => 4723,
    desired_capabilities => $caps
);

use DDP;
p $android;
sleep(60); # optional, if you want to see it open on your device

The $caps object needs to know where your apk is, as well as the app-package and app-activity that we saved from earlier's ionic platform add android. The browserName key is a holdover from some old Appium dependecies, and platformName is straightforward. There's more information about the appium caps in their documentation.

Remember to point your bindings at the correct port - by default, appium starts up on 4723. Pass in your capabilities object, and you should be good to go - running that script with my Galaxy Nexus plugged in on USB debugging mode opened the app with the default Cordova splash screen. Without that last sleep, the app pops open and closed too quickly to notice, but of course you can remove it when writing real test scripts.

To emulate the Android app, there's some useful information about how to get it working if you run ionic emulate android in the toy project directory. ↩︎

An AngularJS File Tree (with a PHP backend)

2014-04-25T00:00:00Z

Something I needed in my AngularJS app was a file tree that exposed a few folders full of flat files to the user. At the beginning of 2014, there were only a few google groups posts about trees in angular - apparently the recursive nature of the tree structure made it complicated to express in the form of directives. Luckily, there are now a few competing file tree projects available. The one I ended up going with is wix's angular-tree-control for a few reasons. I'll go over a few of those reasons, walk through the frontend usage, and also detail the backend PHP code that generates the tree in the right format that angular-tree-control can appropriately consume.

[[MORE]]

1. reasons

I went with angular-tree-control primarily because its light theme looked the best in that it closely matched the old filetree that this one was replacing. Most of the other projects weren't themed appropriately (or at all) for my project. Although I have a decent grasp of CSS, I'd much rather leave the styling to some one else and save myself the time. Additionally, it had the valuable built-in option of clicking on the label to expand/collapse the children of a node. This again was a constraint that came from the previous implementation, and I was aiming to replicate the experience as much as possible so as not to disrupt my users during the transition. Finally, the documentation for this project was very good, complete with a great example page that mimics the official AngularJS docs. And, as an added bonus, it's still under current development, with the author merging PRs in a timely fashion and recently making a few new commits last week.

2. front-end usage

So. Getting the tree into place is pretty straightforward:

<treecontrol class="tree-classic"
             tree-model="tab.data"
             options="treeOptions"
             on-selection="tree.show(node)">
  {{ node.label }}
</treecontrol>

treecontrol is the name of the directive that we can drop into our html template. The class determines its theme; tree-model indicates where on the scope our model. options is the object on our scope that specifies the options, and on-selection is our on-click handler. Finally, the content of the <treecontrol> element becomes the label for each node.

Each node is of the format

{
    label: "aBranch",
    children: [
        {
            label: "aLeaf",
            children: []
        }
    ]
}

If the children array is empty, then the node is displayed as a leaf/file; otherwise it's a branch/folder.

Getting into our controller, we have a few things to set up:

Prevent Selenium Webdriver's Firefox from stealing focus on OS X

2014-05-03T00:00:00Z

When I'm composing automated functional browser tests, I usually run the tests against a local instance of Webdriver on my own computer to keep the feedback loop tight. Once authoring is done, I'll kick it off to Saucelabs for cross-browser testing and additional validation.

On OS X, if I instantiate a new Chrome webdriver browser, it politely stays in the background unless I go looking for it, so I can watch and debug my test output on the CLI. Unfortunately, when Webdriver starts a test in Firefox, it steals cursor focus, shows up on top of whatever windows I already have, and interrupts my workflow.

There's apparently a plist option that solves this very problem:

<key>LSBackgroundOnly</key>
<string>True</string>

By adding that to the top of my /Applications/Firefox.app/Contents/Info.plist file in the <dict> tag, Firefox no longer steals focus. (You may or may not have to sudo to edit that file...). Previously:

<plist version="1.0">
<dict>
    <key>CFBundleDevelopmentRegion</key>
    ...

and now,

<plist version="1.0">
<dict>
    <key>LSBackgroundOnly</key>
    <string>True</string>

    <key>CFBundleDevelopmentRegion</key>
    ...

In fact, with that key in the plist, Firefox doesn't show up in the Alt-Tab menu, the dock, or in the Force Quit Applications window either! When I try to focus on it, it hides behind all other windows. I ended up having to kill it via command line, which was nice since I'm trying to get more comfortable with awk.

$ ps aux | grep [f]irefox | awk '{print $2 " "}' | xargs kill -9

Seeing as it's a pretty generic plist solution, it probably works for other apps too. But, it will affect all instances of Firefox, not just the Webdriver one, so it's possible this may not be the right solution if Firefox is your daily browser.

Trimming and down-casing strings with AngularJS Filters: lowercase and limitTo

2014-05-04T00:00:00Z

I've got a small ng-repeat over this array:

$scope.tabs = [
    { label: "Features" },
    { label: "Phrases" },
    { label: "Sets"}
];

I need the labels to be capitalized in at least one place, as I'm using them as tab headers via UI-Bootstrap's Tab component. But, I'd like to also use the same text lowercased and in their singular forms, so I can have placeholder text like "search for your feature files". Enter the lowercase and limitTo filters!

lowercase does pretty much what you'd expect:

{{ "Features" | lowercase }} => features

and on strings, limitTo:integer trims the string length to the specified integer argument. I tried putting an expression in as the argument to limitTo on a whim, and it worked wonderfully (plnkr):

{{ "Features" | lowercase | limitTo: "Features".length - 1 }} => feature

You can of course replace "Features" with a variable on your scope to make that more versatile - my expression with the array above ended up being:

{{ tab.label | lowercase | limitTo: tab.label.length - 1 }}

N.B. - make sure you've got your scope inheritance sorted when you're using ng-repeat - bind to objects, not primitives :)

Cleaning up whitespace in directories with some elisp

2014-05-05T00:00:00Z

I had a project with an outdated indentation convention. Present-me recently decided to change bracket conventions from the way past-me had been doing things. So, I updated my emacs settings so that magnar's (cleanup-buffer) function would properly format the files. But, there were multiple files in the project and I didn't feel like manually opening each one just to cleanup (C-c n) and save (M-s) it. So, I had emacs do it for me, relying heavily on the excellent f.el library and the aforementioned (cleanup-buffer).

(require 'f)
(defun cleanup-in-directory (directory &optional match recursive)
  (let* ((matcher (if (not (eq nil match))
                      (lambda (file) (string-match match it))
                    nil))
         (files (f-files directory matcher recursive)))
    (mapc (lambda (file)
            (ignore-errors
              (find-file file)
              (cleanup-buffer)
              (save-buffer)))
          files)))

(cleanup-in-directory "/opt/honeydew/lib" ".pm$" t)
(cleanup-in-directory "/opt/honeydew/bin")

It takes a directory, a regex to optionally match against, and a flag to determine whether or not to recurse the parent directory. Small problem, small elisp: problem solved :).

I did glance at the changes before committing them, and caught one case where the formatter got confused and stopped indenting things entirely, but there was only one such case that I noticed...

Cannot open load file: subst-ksc - os x 10.9.2, emacs 24.4.50.1

2014-05-17T00:00:00Z

I'm on OS X Mavericks 10.9.2, running Emacs 24.4.50.1 GNU Emacs 24.4.50.1 (x86_64-apple-darwin13.1.0, NS apple-appkit-1265.19) compiled via homebrew. I was trying to get back into Emacs package authoring with the help of rejeep's excellent Cask and ert-runner tools, but I was getting an error that didn't pop up on Google very well:

bash-3.2$ cask exec emacs -batch -l test/grunt-test.el
Cannot open load file: subst-ksc

Searching for subst-ksc on google pops up results from other OS X users from two or three years ago and no obvious fixes. I tried a couple different things and ended up putting a fake subst-ksc.el file in my load path with (provide 'subst-ksc) in it just to push past the error, but that just gave more of the same:

bash-3.2$ cask exec emacs -batch -l test/grunt-test.el
Cannot open load file: subst-gb2312

Eventually, I realized via $ emacs --version that I recently switched to a new computer where I didn't symlink/replace the default /usr/bin/emacs binary with my newer homebrew'd version. I tried the new method recommended in the comments of replacing that file with a short shell script to invoke the proper emacs:

#!/bin/sh
/Applications/Emacs.app/Contents/MacOS/Emacs "$@"

and all became well.

Playing with the new Perl bindings for the BrowserMob Proxy

2014-05-24T00:00:00Z

Browsermob Proxy is an open source project that Patrick Lightbody split off from the main Selenium project. It works really well in tandem with Selenium Webdriver, but it was missing Perl bindings. BMP exposes a RESTish interface for interacting with the proxies, so I've strung together a Perl module to take care of it: Browsermob::Proxy. Amongst other things, you can use BMP to throttle net traffic during tests, analyze request/response pairs for things like Omniture and Google Analytics, and even alter the requests on the fly with custom headers. Here's a short run down and some basic get-started scripts for using it!

[[MORE]]

The bindings are up on CPAN with pretty limited functionality at the moment - definitely not the full BMP API. It can create proxies and HARs, and spit out the HARs back out when requested, but it doesn't do much else besides that as of v0.04. To get up and running with BMP, you'll need to download the binary from the BMP website and then execute the appropriate file in its bin/ folder to start the BMP server.

Here's an example of using Browsermob::Proxy with Selenium::Remote::Driver to capture all of the network traffic during a single request to google:

#! /usr/bin/perl

use strict;
use warnings;
use DDP;
use Browsermob::Proxy;
use Selenium::Remote::Driver;

my $proxy = Browsermob::Proxy->new(
    server_port => 8080,
);

my $driver = Selenium::Remote::Driver->new(
    browser_name => 'chrome',
    proxy        => $proxy->selenium_proxy
);

$driver->get('https://www.google.com');

p $proxy->har;

This is assuming your BMP server is already downloaded and currently running on port 8080. But, that's all we need to create a new proxy for the Chrome browser that Selenium::Remote::Driver is about to make. The $proxy comes with a convenience method selenium_proxy() that returns the appropriate desired capabilities to configure the proxy for use with Chrome. (If you want to use a proxy with Firefox, you'll need to use a custom Firefox profile.)

Once we've instantiated our driver, having told it to use the proxy, all that's left is to create some traffic and spit out the HAR.

If you want to try out the module without having to install Selenium::Remote::Driver, you can just curl some traffic across the proxy:

#! /usr/bin/perl

use strict;
use warnings;
use DDP;
use Browsermob::Proxy;

my $proxy = Browsermob::Proxy->new(
    server_port => 8080,
);

$proxy->new_har;
my $generate_traffic = 'curl -x http://127.0.0.1:' . $proxy->port
    . ' http://www.google.com > /dev/null 2>&1';

`$generate_traffic`;
p $proxy->har;

This creates a proxy in exactly the same fashion and then uses curl to generate the traffic.

Upcoming updates for Selenium::Remote::Driver

2014-10-20T00:00:00Z

We're gearing up for the release of v0.22 of Selenium-Remote-Driver, a set of Perl bindings for the Webdriver project. My primary motivation to get a new release out is to tweak some of the classes to be more easily extended, so that the perl bindings for Appium can be written more cleanly. But, there's also some cleanup for deprecated functions and a few bugfixes included, along with plans for another release in the near future.

Before we get in to the changes, here's a heads up about the new mailing list for the perl bindings: https://groups.google.com/forum/#!forum/selenium-remote-driver :D

[[MORE]]

A new subroutine get_user_agent will be making it into the API for Driver.pm as a convenience method to, well, get the user agent!
The main changes for Appium were allowing for more dependency injection of classes like ::RemoteConnection and ::ErrorHandler. This lets us tweak things in the Appium module to address discrepancies between Appium and Webdriver. This is also the case for the FINDERS constant defined in the ::Driver class - previously, the constant was privately scoped to Driver.pm, preventing modification from Appium.pm. But Appium has a different list of acceptable finders. So, ea68e0 makes it so the Appium class can declare its own FINDERS constant via the use constant pragma.

Thanks to a few updates to Dist::Zilla::Plugin::TravisYML, our Travis tests are passing again on the master branch instead of just on cpan - there used to be test failures just getting Dist::Zilla to install, even though our own tests were passing. The most recent version of ::TravisYML fixed those issues!
peroumal1 caught a couple deprecated functions that will now be throwing warnings - get_speed, and set_speed are already no-ops and now will be warning appropriately. The same goes for the drag subroutine in ::WebElement, which no longer seems to be in the (not actually official) JSONWireProtocol.
A few bugfixes made it in, including more helpful error messaging when failing to initiate a session on Saucelabs, or when chromedriver is incorrectly configured. There was also a casting issue reported that I still can't reproduce for setting window size where the JSON conversion was using strings instead of integers ("1280" x "1024" vs 1280 x 1024). I still can't reproduce the issue, but I can't imagine the fix to cast the variables into integers causing any other problems, so I'll probably include it anyway.
On the horizon, we've got a rewrite of our recording/mocking functionality on the way, courtesy of peroumal1. Looking over the code in their mock-driver-experiment branch, things seem to be shaping up quite nicely. This is a welcome change because we've been dealing with a complicated process and an explicit dependency on LWP::Protocol::PSGI v0.04 to generate our recordings, effectively preventing anyone else from contributing recordings.
Speaking of release versions, this is the numbering scheme I'm trying to use. The impression I've gotten is that CPAN version numbering is a bit of a grab bag, so perhaps explanation is in order.

A point increase for deliberate releases with new/important functionality - ie, 0.21 to 0.22.
A minor version increase for bugfixes - 0.21 to 0.2101 to 0.2102, etc.
Dev releases to prepare for the next point release go out on the 0.XX50 namespace and increment - so, 0.2150, 0.2151, etc.

drag'n'drop with Perl, Webdriver, and Selenium::Remote::Driver

2014-11-23T00:00:00Z

(This post is direct language translation of ep 39 of Elemental Selenium, Dave Haeffner's Ruby series about Webdriver. Go check 'em out, even if you don't use Ruby, as Webdriver concepts are the same across bindings. We're even reusing his fixture code, as it's quite useful!)

Although the webdriver protocol offers a few endpoints that allow you to accomplish drag and drop (button_down, mouse_move_to_location, and button_up), there's an outstanding bug in webdriver for those endpoints on HTML5 pages. So, there's nothing we can do from the side of the Perl bindings, as invoking the endpoints won't actually do the drag and drop. Luckily, there's a javascript workaround that allows us to simulate drag and drop events with jQuery (or Zepto), courtesy of Rob Correia's gist.

[[MORE]]

First things first, you'll need the helper javascript downloaded somewhere - you can use __FILE__ or FindBin to locate it relative to your perl script. Assuming you save the helper js as drag.js in the same directory as your test script, your test might look like:

#! /usr/bin/perl

use strict;
use warnings;
use FindBin;
use Selenium::Remote::Driver;
use Test::More;

my $d = Selenium::Remote::Driver->new;

$d->get('http://blog.danielgempesaw.com/post/103347925809');

my $grab = $d->find_element('column-a', 'id');
my $target = $d->find_element('column-b', 'id');

my $drag_file = $FindBin::Bin . '/drag.js';
open (my $fh, "<", $drag_file);
my $drag_js = join('', <$fh>);
close ($fh);

my $simulate_js = '$(arguments[0]).simulateDragDrop({ dropTarget: arguments[1] })';
$d->execute_script( $drag_js . $simulate_js, $grab, $target);

is($grab->get_text, 'B', 'text in column-a has changed!');
is($target->get_text, 'A', 'text in column-b has changed!');

$d->quit;

done_testing;

This is accomplishing exactly the same thing as the Ruby version of the script. We load up our driver, get to our test page, and then we're deviating slightly. I prefer to find the elements in webdriver separately and pass them into the execute_script, as my framework allows users to locate elements in various with multiple by strategies, so we'd need account for that deviation. Luckily, execute_script puts any additional arguments it receives into the special arguments array in javascript, transforming them to and from actual DOM elements if necessary. So our $grab and $target are available in their DOM element form as arguments[0] and arguments[1], respectively.

The final step is to concatenate the tiny drag library with the javascript to simulate the drag and drop. After that, we just verify that the text in the columns changed, and we're good to go!

I'm wondering if I should include the $drag_js in Selenium::Remote::Driver somewhere. We recently found a deprecated drag method on our WebElement class - since the drag endpoint no longer exists in the JSONWireProtocol, we've begun the process to deprecate it in our module. However, the aforementioned workaround means that we could restore our binding's drag method with this hack until an official fix comes in. But, the official fix might lead to a change in API that would confuse our users.

On the one hand, it always feels a bit icky to include javascript in a perl module. On the other hand, drag and drop is pretty commonly requested and it could be useful to many users. I think I'll leave it out of our library until there's an explicit need for it, and point people to this method until then.

Drag and Drop Test Fixture

Using SIPE through Bitlbee in Emacs on OS X Mavericks

2014-12-10T00:00:00Z

I like using Emacs as often as possible. Our work server was previously jabber, so I was using the excellent jabber.el to chat with my coworkers. We're switching over to Microsft Lync, so it appears my best option is to use Bitlbee and an Emacs IRC client (like Circe!). Bitlbee is a server that you can run on your computer that acts as a gateway to different chat protocols. It handily exposes your IM interactions to you through an IRC server. One of the protocols it supports is SIPE, which is what Microsoft Lync uses. It was slightly involved to get all the pieces working together; here are the steps I followed.

[[MORE]]

1. get pidgin and bitlbee

$ brew update
$ brew install bitlbee --with-pidgin

--with-pidgin specifies the pidgin package as a dependency, which I believe we'll need to manually compile pidgin-sipe, since it's not available on homebrew. In case brew link bitlbee doesn't put bitlbee in your path, you can just

$ ln -s /usr/local/Cellar/bitlbee/3.2.2/sbin/bitlbee /usr/local/bin/bitlbee

2. get pidgin-sipe

pidgin-sipe will teach bitlbee how to speak SIPE! Download the latest tarball from the sipe project on SourceForge. At time of writing, the direct download link to 1.18.4 is here. Compiling it requires gettext, and it may not be linked by default, so link it if necessary:

$ brew link --force gettext

Unzip the pidgin-sipe.tar.gz tarball, get in that folder, and get to work compiling it:

$ ./configure --disable-quality-check --prefix=/usr/local/
$ make
$ sudo make install

The -prefix option apparently is used to tell pidgin-sipe where to find the libpurple files that we installed as part of pidgin during the earlier bitlbee --with-pidgin step.

3. tell bitlbee what's up!

Once you've got pidgin, pidgin-sipe, and bitlbee all installed, you should be able to get started playing around with bitlbee (finally!). You'll want to run it in ForkDaemon mode via -F, as it suggests as a consequence of compiling --with-pidgin.

$ bitlbee -F

Fire up your IRC client of choice and connect to localhost:6667 and join the control channel "&bitlbee"^[1]. Say help purple in the &bitlbee channel to see your potential protocols, and hopefully sipe (Office Communicator) will show up in the list! If it's missing, you'll need to go back and verify your dependencies. Forging bravely on, add an account with the sipe protocol - although the Bitlbee wiki says to include DOMAIN\LOGIN for the account add sipe command, I left it blank and had no troubles logging in. YMMV.

account add sipe EMAIL@ADDRESS.COM PASSWORD
account sipe set authentication tls-dsk
account sipe set useragent "UCCAPI/15.0.4481.1000 OC/15.0.4481.1000 (Microsoft Lync)"
account 0 on

There were a final few things to tweak: we needed to set the authentication scheme (TLS-DSK) and masquerade as an actual lync client by setting our user agent^[2], but after that, we get the golden output:

<root> sipe - Logging in: Connecting
<root> sipe - Logging in: Logged in

Use blist all to see your verbose buddy list, and chat away! Ta-da!

Honestly, I'm not very good at IRC so it might bring up that channel automatically, or something - I'm autojoined to it in Circe, at least. You can optionally register an admin account to persist your settings via bitlbee's register, and subsequently identify, commands. ↩︎
And perhaps we need to set an environment variable? I didn't need to, but obviously some people did: http://blog.mattwoodward.com/2012/08/pidgin-sipe-and-read-error-on-ubutnu.html ↩︎

Getting Started with the Appium Perl Bindings

2015-01-08T00:00:00Z

We've started work on a set of Perl client bindings for Appium. We're using Appium to test out a few different ideas for mobile apps, and since our existing browser automation framework is written in Perl, it makes sense to use Perl for our mobile automation as well. I recently received an email from probably the first person other than myself interested in such a pairing (Perl & Appium) for some help getting started, so I'm putting together a collection of steps to get going with the Appium Perl bindings.

1. Install Appium

You can install Appium via npm:

> npm install -g appium

It comes with a helpful binary appium-doctor that will give you the status of your setup. The Appium project has official installation docs, and I made a previous post with OS X steps on this blog a few months ago. This post will be slightly more Windows focused, but it's all a bit theoretical, as I'm not using a Windows machine to write this post :P. For running Android tests specifically, you'll need to download the Android SDK Tools, and set/modify a couple environment variables; we've broken each part down into the following steps.

2. Download Android Dependencies

Follow the instructions on the Android Developer website for downloading at least the SDK Tools. After the download, you'll need to open the SDK Manager and get the appropriate tools that correspond to the version of Android you're interested in; the Android developer website has a short guide on how to do so.

From the Tools folder, you definitely want the SDK Tools, the Platform-Tools, and the Build-Tools. Choose the highest version of Android, or whatever version of Android your device has, and select at least the SDK Platform and a system image like ARM EABI v7a System Image. My SDK Manager looks like this with those checked. Select at least those, click Install packages... and accept all the licenses for those packages.

3. Set up Android Environment Variables

You'll need to set JAVA_HOME if it's not set; Atlassian has a nice article about doing that. You'll need to do the same thing for ANDROID_HOME: find the path of where you downloaded the Android SDK, create a new environment variable called ANDROID_HOME and set it to the download path of the sdk.

Finally, you need to add the tools/ and platform-tools/ directories inside of ANDROID_HOME to your PATH. It will be in the same place as when you edited JAVA_HOME and ANDROID_HOME; choose to edit the PATH variable and add ;E:\android-sdk\tools;E:\android-sdk\platform-tools to the end, assuming of course that your SDK was in E:\android-sdk\; adjust as necessary.

4. Build or obtain an .apk of your app, or pre-install it

This will vary wildly depending on the app you're building and your particular build tools. Build your app and get a hold the path to your .apk. You also have the option to pre-install the app on the emulator or device on which you will be testing.

5. Make and start your Android emulator

If you're using a real device, plug it in and allow the USB debugging permissions when prompted on your device. You may need to install drivers, or perhaps Windows is smart enough to do it for you - I can't quite recall.

Alternatively, you can create and start an emulator with the following commmands:

> android create avd --force -n appium -t android-21 --abi x86
> emulator @appium

That will create an Android Emulator called "appium" for the latest version of android (android-21/lollipop). The second command will start it up; you should see it pop up on the screen.

At the end of this step, doing adb devices in a command prompt or console should get you some output like the following:

> adb devices
List of devices attached
emulator-5554   device

If you're using a real device, it won't say emulator. If the output of adb devices is blank, something went wrong :(

6. Write a perl test!

At this point, you should be able to run appium-doctor in a command prompt without any complaints; if you see some, you should go back and fix any outstanding issues. Hopefully all checks are successful; if so, kick off the Appium server and keep reading.

> appium

To install the Perl bindings, you can use CPAN or cpanm:

> cpanm Appium

With your emulator or device connected (verify via adb devices) and Appium server running (verify via wget localhost:4723/wd/hub/status), the following script should get you started after you fill in the path to your apk.

use strict;
use warnings;
use Appium;
use Test::More;

my $path_to_apk = '';
my %caps = (
    app => $path_to_apk,
    deviceName => 'Android Emulator',
    platformName => 'Android',
    platformVersion => '5.0.1',
);

my $appium = Appium->new(caps => { %caps });
ok($appium, 'We have an Appium driver available!');
$appium->quit;

done_testing;

At this point, it's up to you to write your tests! :) If anything doesn't work, check the Appium log for warnings and errors, especially about ANDROID_HOME or any of the environment variables we set up previously.

Feel free to reach out with any issues; if you come across bugs in the Perl client bindings, its Github repository is waiting for its first bug report! Good luck!

Using Emacs for daily reminders to write in my ~/diary

2015-02-01T00:00:00Z

I worked remotely for a year or so and I took it upon myself to figure out ways to improve my communication with HQ. One of the useful things I found was sending a weekly email summary to my manager, as a touch point for them to be able to see my week to week progress^[1]. I soon realized that I had no memory at all for recalling what I did the previous day, much less for a whole week, so I began looking into what options Emacs offered for daily diary note-keeping kind of things^[2].

[[MORE]]

Google told me that I should to use the calendar and diary functions to make my notes. M-x calendar brings up a three month calendar view, and M-x diary opens the ~/diary file on your computer in a special diary-mode. Each line in the diary file starts with the date in whatever format you want ('iso), and then you can put notes for that day. When cursoring about the calendar, pressing i d opens up the diary with a new entry for that date ready to accept some text.

;; dg-diary.el
(require 'calendar)
(calendar-set-date-style 'iso)


;; ~/diary
2015-01-17 some notes for the day!

The diary and the calendar have a pretty tight integration that I'm still trying to learn! Of course, the biggest issue was remembering to actually remember to write in my diary in the first place. For a while, I had a Google Now reminder, but a daily reminder ended up clogging the entire Inbox interface, as they really call reminders to the forefront there. So, I tried to figure out how to set a daily reminder in Emacs and came up with the following:

(defun remind-me-daily (fn time)
  (when (and (boundp 'daily-reminder)
             (timerp daily-reminder))
    (cancel-timer daily-reminder))
  (let ((daily (* 60 60 24)))
    (setq daily-reminder
          (run-at-time time daily 'funcall fn))))

(remind-me-daily 'toggle-diary-windows "4:30pm")

Basically, run-at-time is doing all the work here. There's a little work to make sure I'm not setting tons of reminders by automatically clearing the existing daily reminder. Otherwise, we just handle the number of seconds in a day and set our function to be called daily.

toggle-diary-windows is a function that stores my current window configuration before bringing up the calendar and the diary, putting my cursor at the correct spot in the diary file to make notes for the current day.

(defun toggle-diary-windows ()
  (interactive)
  (toggle-app-and-home
   "Calendar"
   (lambda ()
     (window-configuration-to-register 6245)
     (calendar)
     (delete-other-windows)
     (text-scale-adjust 3)
     (execute-kbd-macro [?m ?. ?i ?d])
     (text-scale-adjust 0)
     (text-scale-adjust 2)
     (recenter-top-bottom))))

toggle-app-and-home is a custom function that checks whether any of my open buffers match its first argument and decides if it should show me the app (Calendar, in this case) or my previous window configuration ("home"). A little involved, but probably easier to understand in person. Basically, I've got a couple different apps that I use in Emacs that are completely separate from authoring code: diary things, mu4e for my mail, jabber for chat, etc. At any point, it's useful to be able to toggle my whole window configuration to and from setups that allow me to do those tasks.

After penning quick summary of the day's work, I use my diary keybinding to toggle back to my previous window configuration and continue on my way. At the end of the week, I've got a few sentence fragments for each day and an easy way to write the Accomplishments, Blockers, and Next Week sections of my weekly summary.

If you liked this post, you may also be interested in how Sacha Chua is doing her process journaling!! My complete file for customizing my diary interactions is on Github.

This was in addition to daily standup check-ins and weekly 1 on 1s and so on - I wanted to do more communication on top of all that. ↩︎
This is unfortunately not an org mode post, as that's one of the Emacs things that I have actually managed not to do so far. ↩︎

Running perl's Test::More tests in parallel on OS X

2015-02-21T00:00:00Z

Integration tests and e2e tests are very useful when developing apps, but unlike unit tests it's harder to keep them fast and snappy. In my case, I have to spin up multiple webdriver instances to test my webdriver automation framework as part of the e2e process. Although webdriver is now significantly faster than it used to be, running twenty tests in parallel is an order of magnitude quicker than running them all in series. I've started using Test::ParallelSubtest to achieve this speedup, due to trouble installing Test::Parallel on OS X.

[[MORE]]

Previously, Test::Parallel was my perl package of choice for dropping into a normal Test::More file and getting an immediate speed up. But at some point a few OS X versions ago, I tried reinstalling Test::Parallel to no avail. Sys::Statistics::Linux::MemStats is failing its configuration because OS X/darwin isn't the intended platform.

bash-3.2$ cpanm -nf Test::Parallel
--> Working on Test::Parallel
Fetching http://www.cpan.org/authors/id/A/AT/ATOOMIC/Test-Parallel-0.20.tar.gz ... OK
Configuring Test-Parallel-0.20 ... OK
==> Found dependencies: Sys::Statistics::Linux::MemStats
--> Working on Sys::Statistics::Linux::MemStats
Fetching http://www.cpan.org/authors/id/B/BL/BLOONIX/Sys-Statistics-Linux-0.66.tar.gz ... OK
Configuring Sys-Statistics-Linux-0.66 ... N/A
! Configure failed for Sys-Statistics-Linux-0.66. See /Users/dgempesaw/.cpanm/work/1424475457.49669/build.log for details.
! Installing the dependencies failed: Module 'Sys::Statistics::Linux::MemStats' is not installed
! Bailing out the installation for Test-Parallel-0.20.

and in the build log

Configuring Sys-Statistics-Linux-0.66
Running Build.PL
OS unsupported! Sorry, but this system seems not to be a linux system! at Build.PL line 5.
Running Makefile.PL
OS unsupported! Sorry, but this system seems not to be a linux system! at Makefile.PL line 3.

But, like previously mentioned, Test::ParallelSubtest installs just fine. It does take a little change to the test code itself, as it requires you to organize the tests in subtests.

One small issue I ran into was keeping my tiny webserver alive for the entire duration of the test. I have to start up the webserver, wait for it to be ready to accept connections, and then I can kick off the tests that hit the server. Since the server I was using automatically cleans itself up during destruction, I got bit when the main process was finishing rather quickly - all it needed to do was put the tests together, and then quit. Meanwhile, the background subtests had barely gotten started.

Luckily, bg_subtest_wait() solves this problem for me. The docs do say that it gets called implicitly along with done_testing, but it wasn't working for me. Manually invoking it at the end of the script kept the parent webserver process alive while all of its kids played in its server sandbox.

Imitating live reload in Emacs for Perl (or any) project :)

2015-03-18T00:00:00Z

One of my favorite parts about my frontend JS workflow is using Live Reload with Grunt/Gulp. Saving a few keystrokes a couple times a minute is pretty great, especially if I can stay on the home keys. Grunt and Gulp have a task that watches the files in your project and then you kick can off certain tasks when they change, like refreshing your display and re-running all the unit tests. In Scala, you can get a similar thing by invoking your sbt command prefixed with a ~, but I didn't have anything for my Perl projects (not to mention a general solution!).

At first, I looked for a similar tool that would watch files for changes and let me hook into that action. I found App::Prove::Watch which was a Perl-only solution. It worked swimmingly after a quick PR, but it only solved the problem for Perl projects. It wasn't until a few days later that I realized I could leverage Emacs to get a reasonably general solution, only depending on use of the *compilation* buffer (no idea why it didn't occur to me earlier!).

Of course, Emacs knows exactly when my files change, as we always eventually invoke (save-buffer) - even if it's advised with with whitespace cleanup, or if it's invoked via some sort of autosave. Instead of watching files on disk for changes, we can just use Emacs to have the (save-command) trigger our tasks - aka re-compiling the *compilation* buffer.

I used a global variable ar-auto-recompile to turn autocompilation on and off, and made a function to handle the toggling for me:

(setq ar-auto-recompile nil)
(defun ar-auto-re-compile ()
  (interactive)
  (setq ar-auto-recompile (not ar-auto-recompile))
  (message (format "Auto recompile is now %s"
                   (if ar-auto-recompile "ON" "OFF"))))

It just toggles the variable and messages us about it. Next, our advice to save-buffer should check whether the user wants auto-recompilation. I also decided I only wanted it to re-compile if the *compilation* buffer was in a visible window, which (get-buffer-window) happily tells us.

(defadvice save-buffer (after ar-auto-recompile activate)
  (when (and ar-auto-recompile
             (get-buffer-window "*compilation*"))
    (set-buffer compilation-last-buffer)
    (revert-buffer t t)))

This solution isn't quite the same as the watch behavior of the other tools, as they smartly only respond to changes in files in their own project. When turned on, this would blindly recompile any time a buffer is saved and the *compilation* buffer is open. But, I don't really mind - a few extra compilations probably won't hurt anything unless the compilation wasn't safe to repeat in the first place.

So, my workflow is now to open up my test file, toggle on the recompilation with (ar-auto-re-compile), run a test to bring up the *compilation* buffer. I keep that buffer visible and then do edits and such as usual, saving with M-s, which automatically re-runs the test. And, if I need a more complicated compilation command, it's just a C-u M-x compile away, and then that can get repeated for me. As an extra bonus, this works perfectly with projectile's compilation and test running commands (by default on C-c p c and C-c p P), as they utilize the *compilation* buffer as well. I also made a key-chord to switch the current buffer over to the compilation one, as I often accidentally close it:

(key-chord-define-global "vv" (lambda () (interactive)
                                (switch-to-buffer "*compilation*")))

Finally, if you don't want to use advice, it's simple enough to use a wrapper around (save-buffer) and update your save keybinding to use your own function. And as usual, the whole snippet is in my github somewhere.

Selenium::Remote::Driver@0.25: now with slightly less JRE

2015-04-20T00:00:00Z

We've just released version 0.25 of Selenium::Remote::Driver to the various CPANs. The big push this time around was to get around our hard dependency on the JRE. Previously, the Perl bindings demanded a standalone selenium server be operating on the browser's machine. So, if you wanted to run tests on your own box, you'd need the Java Runtime Environment installed, as the selenium-standalone-server is a .jar and needs the JRE to execute. However, as akafred pointed out, this is a prohibitive constraint (perhaps especially so for Perl programmers?).

When I first started out working with Webdriver years ago, I only used the standalone server and for a while I thought there was no other way to run the tests. Eventually, when various webdrivers began replacing Selenium RC, I found out that it was possible to "talk" directly to them with the same exact API as the standalone server, but I never connected that with the ability to avoid needing the standalone server and the JRE!

Anyway, thanks to some long-awaited prodding, Perl can now run Firefox, Chrome, and PhantomJS without the JRE! What follows are some implementation details, and simple usage examples. :)

[[MORE]]

usage

Hopefully the usage is pretty straightforward - instead of constructing a Selenium::Remote::Driver instance, use the constructor for the browser of your choice instead. Just like the Selenium standalone server, Firefox will work out of the box, as long as you have Firefox installed locally on your machine in the default location:

my $firefox = Selenium::Firefox->new;
$firefox->get('http://www.mozilla.org');

# "We’re building a better Internet — Mozilla"
print $firefox->get_title;

The usage is exactly analogous for the ::Chrome and ::PhantomJS classes, except you need to have the browser installed along with its associated webdriver. For PhantomJS, GhostDriver is automatically bundled for all recent versions, but for Chrome, you'll need to download the Chromedriver separately.

If your driver executables are not in your $PATH, or you'd like to specify a certain one, the constructors take a binary option that lets you tell us what executable to start. You can also specify a binary_port if there's a specific port that you'd like the webdriver server to bind to.

my $firefox = Selenium::Firefox->new(
    binary => '/custom/path/to/firefox-bin',
    binary_port => 65432
);

As expected, usage is analogous for the other two classes.

As a last note, we skipped creation of Selenium::InternetExplorer hoping that YAGNI, but if that's the browser that floats your boat, we can definitely throw a class together for the next release. Let us know in our Google group, or in the Github issues!

implementation

The implementation is ideally pretty straightforward:

Find the binary webdriver
Figure out what arguments to pass it to make it mimic a standalone server
Start it on the right port
Afterwards, clean it up so we don't orphan processes

The first step is easy - just check the $PATH/%PATH% - exceeept for Firefox, where it's apparenty quite complicated. Different operating systems install it in multiple distinctly different places, and if different versions of Firefox are named differently - basically it's seems like a big headache, and I mostly threw my hands up if the Firefox binary wasn't in the first specific default location. There's also a bit of extra complication involved to let the user choose their own path, and validating it for them afterwards.

Passing the arguments is more or less straightforward as well - exceeept for Firefox, which doesn't take arguments, as the webdriver for Firefox is actually just a Firefox extension. Luckily, we previously implemented a Firefox::Profile class, so we just use the newly renamed Selenium::Firefox::Profile to create a profile with the extension loaded. We actually now bundle the webdriver.xpi extension from the 2.45.0 version of Selenium in our release, and each time a new Selenium is released, we'll have to do a mirror release of our bindings with the updated extension. The other webdriver language bindings also start up Firefox with a few pre-compiled .so files for solving focus errors that I was also too lazy to do.

Finding an open port is simple enough, and we got to re-use Selenium::Waiter's wait_until in a few places. And, starting the webdriver up is usually easy, although at first I was a bit unfamiliar using system to start up an asynchronous process across different platforms (namely Windows...).

Finally, cleaning up the process we started is straightforward on OS X and Linux, as I think it just cleans itself up when the Perl script ends. At least, that's what some superficial tests seemed to prove - I didn't see anything in ps aux after the test was over, and I couldn't open sockets to the server port afterwards. But, on Windows, the task stays open and we need to kill it by string-matching the title of the process. It ended up being a bit of a mess!

We went through a bunch of attempts & refactors at organizing the functionality and ended up at something that's decently organized - the Selenium::CanStartBinary role implements most of the common work between the different classes. Since Firefox is special, it gets a few extra classes of its own. Meanwhile, each class holds its own arguments, default binary name, and default binary port.

conclusion

For a first pass, I think the functionality works pretty decently - as usual, I've been using it locally for a month or two now on OS X with no major issues, but I'm sure I missed a few bugs. If you run across any bugs, definitely let us know or even fix them for us, since I bet you can do it better than me! :D Cheers...

Waiting for a page to load in Selenium::Remote::Driver with Selenium::Waiter

2015-04-21T00:00:00Z

One of the first problems we come up against when trying to use Webdriver to automate a web application is handling the asynchronicity of loading a web page. For the test to be useful at all, it needs to be as fast and reliable as possible, or else there's a very real risk that devs and QA engineers will give up running the tests. The problem is that making the test run faster can sacrifice reliability if the async javascript isn't properly handled.

Instead of trying to figure out when a page loads, we should take a hint from what an actual user does. Users don't care when a page is done loading - they just wait for the exact element they want to interact with, and then start clicking/inputting right away. We can mimic that behavior with wait_until, a utility function exported by Selenium::Waiter.

[[MORE]]

The general case of determining when a page is done loading is a variation of the halting problem. When you ask Webdriver to load a page, it does block briefly while it runs through its own algorithms to figure out when the page is done loading. But, it being the halting problem and all, they obviously can't solve it for all the cases.

my $d = Selenium::Firefox->new;
$d->get($tricky_slow_loading_page);

# this will throw when the element isn't present
my $text = $d->find_element_by_css('div')->get_text

The problem with putting in a sleep before attempting to find the element is that it will usually work, but maybe one time in twenty it will fail, and it won't be immediately apparent why it failed, especially months down the line. What we want is a method that reliably and consistently waits exactly until the element in question is ready. We don't want to wait any longer than necessary, so a long explicit sleep is out, but we don't want to go early, or else we'll get exceptions all over the place. wait_until lets us get pretty close to this ideal behavior:

# wait_until will also catch dies and croaks
my $elem = wait_until { $d->find_element_by_css('div') };

if ($elem) {
    say 'Text: ' . $elem->get_text;
}
else {
    say 'We waited thirty seconds without finding css=div';
}

wait_until takes a block and an optional hashref of arguments. It wraps the block execution in a try/catch from Try::Tiny. By default, it will run for thirty seconds, sleeping one second between iterations. If at any point the block returns something true, it immediately returns that value as its result. Note that wait_until expects a block that is generally NON-blocking, so if webdriver has an associated timeout, like the implicit wait timeout for finding elements, you'll want to set it to a second or less if you've increased it. The exact number of iterations will depend on how long the block takes to execute.

To be clear, if your implicit_wait_time is 31 seconds, and you put a find_element inside a wait_until, we'll run it once, Webdriver itself will block for 31 seconds, and by the time we get control back in our wait_until block, the timeout will have expired, and we'll return control to your script after executing exactly one find_element. (This may be the behavior you desire - but just make sure you're doing it on purpose!)

my $d = Selenium::Firefox->new;
$d->set_implicit_wait_timeout(30000);
my $one_iteration = wait_until { $d->find_element('this is blocking', 'css') };

You can also use wait_until to have your test block until the element is visible, or some other boolean property:

my $visible_elem = wait_until {
    $d->find_element_by_id('eventually-visible')->is_displayed
};

Finally, as mentioned, wait_until wraps everything in a try, so if the BLOCK you passed in does die, it'll get demoted to a warn. This means you MUST check the return value of wait_until. Normally, Selenium::Remote::Driver will croak when we run into something we don't understand ^[1]. Since we're only warning, it's possible you may get into some weird territory if you make assumptions about the return value. Honestly, I'm still not sure about this behavior - perhaps it makes sense for wait_until to die if the expected value never returns true.

Although it can be frustrating, it's helpful for the program to die as close as possible to the source of the crash. Also, from the test's point of view, we have no idea what to do if we get an unexpected exception. ↩︎

The Abuse and Misuse of Test Automation – Interview with Alan Page - Fog Creek Blog

2015-05-28T00:00:00Z

I've been thinking recently that one of the big problems we're having with automation at work is caused by previous-me of years past encouraging the idea of "oh doing automation is easy, look, gherkin is close to readable English, that makes it simple!" However, we're a couple years in to our (perl!) Gherkin & Selenium based automation framework and we're running into a lot of pain points.

What I've realized, perhaps unfortunately a few years too late, is that E2E automation is very complicated to do well & usefully. It takes a deep knowledge of multiple working & changing systems (app under test, the automation framework, and everything in between) to do it effectively. Bringing in people who aren't familiar with either (through no fault of their own) and neglecting to train them properly is a good way to get a brittle automation suite.

I think that's one of the big problems I should try to solve this year. In the linked podcast episode, Alan Page espouses some pretty similar ideas about how to do automated testing well, pointing out that having junior developers write (e2e) tests often leads to failure, where as more senior developers who are comfortable with the complexity of the entire system are actually able to successfully write useful tests:

Another reason, and I’m sorry to say this, is often we have is this, really scary when you think about it, to me at least… We often have our most junior developers, our brand new testers who are learning to code or maybe some company that’s where you start people. You get these very junior people without a lot of design knowledge for software and a lot of experience. You tell them write this GUI automation. You have junior people doing something that’s much, much harder than most people admit and it just makes it more prone for failure.

I think when you get success is you have developers who know what they’re doing, more senior developers. It doesn’t have to be your top developer but you have experienced people who know what they’re doing. They understand design and they’re very careful about writing tests that they can trust to show product status. That’s the case that GUI automation can actually work.

He's also pretty strict about flaky tests, which I unfortunately don't have the luxury to get too hung about. Since I'm the main (aka only) developer on our automation framework, supporting a growing group of QA engineers, I don't have time to dig into every flaky issue. Hm, but, putting it on paper like that, it actually becomes pretty apparent to me that I do need to spend some time digging into flaky issues - otherwise I'll get drowned in flaky bugs that negatively impact our view of the framework. That's already happening, and we do need to actively combat it.

Another question that we're always curious about is what to automate. Although they're not wrong with their "automate all the tests that should be automated," and "automate when you're bored," heuristics, I find it a bit lacking to translate that into specific situations that I can tell our team about.

( also I like this trend of transcribing podcasts! I can read much faster than I can listen :) )

Bitlbee’s account xml file on os x

2015-07-29T00:00:00Z

This is primarily for me to find next time I run into this issue! Having previously installed bitlbee on my OS X machine via homebrew & some elbow grease, I made a one-off password for the local bitlbee server I run to store my account credentials. Since I run the bitlbee server through Emacs and like a well behaved Emacs user, I hardly ever restart Emacs, I'm sure to have forgotten my bitlbee credentials between server restarts. At those times it's pretty useful to be able to find the bitlbee XML file that houses my username and encoded password. This is also useful in case I need to change other account setting things.

So, for bitlbee installed on OS X via homebrew, the account xml file is in the following folder on my machine:

/usr/local/var/bitlbee/lib/<username>.xml

and the bitlbee server wants the credentials like

> identify <password>

My own reference for flashing new android without losing root or user data

2015-08-04T00:00:00Z

Humm, so my nexus 6 has a persistent notification about wanting me to upgrade to 5.1.1. I was lazy this time and didn't bother doing it at the end of May when the stock images came out, but my phone doesn't know how to do it on its own since it's rooted and unencrypted and/or it's using TWRP recovery instead of the stock recovery. I don't do this frequently enough to remember, but too infrequently to want to write a script for it (especially since Wug's toolkit already exists). Anyway, here's my pretty straightforward steps to getting my N6 to 5.1.1, generic enough to apply to any update^[1]:

start downloading whatever factory image that needs flashing
on the phone, make a backup via TWRP
copy the backup to a desktop, just in case of lightning strikes!
whenever the DL finishes, unzip the .tgz and go in that folder
in that folder unzip the .zip, cuz it has boot.img and system.img
wire the phone to a computer (probably already done in 2)
reboot to bootloader (power off, hold vol down & power)

use fastboot to do some things:

 fastboot flash bootloader bootloader.img
 fastboot flash radio radio.img
 fastboot reboot-bootloader
 fastboot flash boot boot.img
 fastboot flash system system.img
 fastboot reboot-bootloader

head into TWRP recovery, supposedly I should wipe cache & Dalvik, but I forgot and it seems fine. anyway, reboot to system and it'll prompt about installing SU for us since we lost root (yay TWRP).

Make sure not to flash the recovery from the download since we'd like to keep TWRP, not the stock recovery. TWRP's reboot into system will spin the "optimizing app M of N" for a while, and then it should be gravy!

I used to go to theSiteThatShallNotBeNamed for this, but after seeing their recent behaviors and reading the thoughts of some industry thought leaders (heh, heh, thought leaders is a funny concept - but here I am following their thoughts...), I figure I'd ought to make a reference for myself so I can stop giving them traffic. ↩︎

Chromedriver and the Weak Ephemeral Diffie-Hellman Public Key

2015-09-04T00:00:00Z

As of Chrome 45, there's a new error message about a weak ephemeral Diffie-Hellman public key that started showing up in our webdriver & chromedriver proxy tests. The intent of the block was to secure users from the LogJam vulnerability.

ERR_SSL_WEAK_SERVER_EPHEMERAL_DH_KEY

In our testing set up at $WORK, we use Browsermob Proxy to MitM our E2E test traffic so that we can analyze the network traffic. Using a proxy allows us to test things like Omniture & Google analytics, and also enables us to simulate XSS attacks against our website.

Our E2E test suite depends heavily on the proxy being allowed to MitM the traffic, and Chrome started noticing that the DH key that our proxy presented was insecure. This is pretty valid for Chrome to want to block, since we are after all attacking ourselves^[1]. Luckily, Chrome allows us to blacklist certain ciphers as an argument during startup and after some wild googling, I arrived upon the following CLI argument for Chrome:

--cipher-suite-blacklist=0x0039,0x0033

[[MORE]]

Basically, we are blacklisting the cipher suites that we don't want to run, and this allows Browsermob::Proxy to go on its merry man-in-the-middle-attack way. Using the perl webdriver bindings to start up a proxy and then chromedriver, this ends up looking like:

use strict;
use warnings;
use feature qw/say/;
use Selenium::Remote::Driver;
use Browsermob::Proxy;

my $proxy = Browsermob::Proxy->new(
    server_addr => $server,
    server_port => 8080
);

my $d = Selenium::Remote::Driver->new(
    desired_capabilities => {
        browserName => 'chrome',
        proxy => $proxy->selenium_proxy,
        chromeOptions => {
            args  => [
                'cipher-suite-blacklist=0x0039,0x0033'
            ],
        }
    }
);

$d->get('https://www.google.com');
say $d->get_body;

Probably the most interesting part^[2] is the particular data structure that chromeOptions requires. The same idea will work with any of the other webdriver language bindings as well - if you can figure out how to pass arguments to your chromedriver via your bindings of choice, just append the cipher-suite-blacklist argment to the list of args.

Also of note - a lot of sources recommended blacklisting a longer list of ciphers:

--cipher-suite-blacklist=0x0088,0x0087,0x0039,0x0038,0x0044,0x0045,0x0066,0x0032,0x0033,0x0016,0x0013

After little trial and error I was able to determine which particular ciphers that I needed to blacklist, so I only chose 0x0039,0x0033, but others may be necessary; YMMV! For further reading, there were a couple very helpful links I found:

Frank Ehlis already figured out how to disable SSL ciphers in Dec 2013.
There's a chromium issue listing all the ciphers.
This superuser also helpfully had the answer about disabling the SSL suites
A diagnostic page for determining which SSL cipher suites your browser supports!

Coincidentally, the security block also to inadvertently caused issues for people with intranet websites that aren't properly secure, amongst other things. This has somewhat understadably ended up angering plenty of internet users who feel very entitled to their free browsers. ↩︎
The other gotcha that I ran into while troubleshooting this issue is that if the Browsermob proxy server was on the same machine as the browser in question, the issue didn't manifest itself at all. We only experienced the issue when the proxy server and the browser were operating on different machines. ↩︎

Stopping footnotes here from opening in a new tab

2015-09-05T00:00:00Z

Tumblr's markdown formatting mode somewhat secretly supports footnotes. But, it seems like my settings or my theme or something makes footnote links with the target="_blank" attribute set, which is pretty odd. Who wants a footnote to pop them to a new tab ? And furthermore, the return links in the footer also have the same target="_blank". Basically, the footnotes on this blog have been nigh unusable, since they keep spawning new tabs all over the place.

So, some quick javascript to get things sorted:

Array.prototype.slice.call(
    document.querySelectorAll( 'a[rel=footnote], a[rev=footnote]' )
).forEach( function (node) { node.target = ''; } );

find the impacted nodes with document.querySelectorAll
convert that NodeList to an Array
clear the target on each node

Honestly, I'm not entirely sure why this happens on this blog - other people don't seem to have the issue. In case anyone else is seeing this behavior on their tumblr footnotes, just add the code above to a <script></script> tag at the bottom of the HTML for the page :)

Fixing a laggy compilation buffer

2015-09-25T00:00:00Z

I use Emacs compilation mode to execute most of my scripts - it has a lot of nice built in things, including error highlighting with automatic navigation. But, I've noticed that with really long lines, the compliation buffer gets so slow that it makes Emacs unresponsive to input, and I end up having to frantically Ctrl+G and hope that I can stop the compilation before I lose Emacs and have to Force Quit it.

One particular instance is when I'm using Webdriver to take screenshots - Webdriver takes the screenshot (in binary format?) and base64 encodes it, so that means I've got a very long string to work with, and if I accidentally print it out, Emacs gets quite overwhelmed.

As usual, I'm not the only one to run into this issue, and there's a gmane.emacs.bugs thread (cache) about disabling the Maven regular expression in the compilation-error-mode-alist:

(setq compilation-error-regexp-alist
      (delete 'maven compilation-error-regexp-alist))

I had thought it had to do with colorization, or perhaps the rainbow delimiters that I use, but that simple line of disabling the Maven regex made a huge difference. Hooray for correcting long-standing annoyances.

Executing promises serially with [].reduce

2017-03-03T00:00:00Z

Recently at $WORK, we were writing a data migration script in node that needed to make a couple hundred requests. The first attempt was just to wrap everything up in a Promise.all:

const rp = require('request-promise-native');
const urls = [
    'url1',
    'url2',
    // ...,
    'url300'
];

Promise.all(urls.map((url) => rp.get(url)
                     .then(sendRelatedRequests)));

[[MORE]]

However, the internal server we were talking to wasn't able to handle all of the requests concurrently, and since the subsequent logic would also send a few more requests of its own, we ended up taking down the server because we were spawning all the promises at the same time, and since Promises execute once they're made, that means all the requests were starting off at roughly the same time.

So, for our second pass, we decided we wanted to only send one request at a time, lining up all of our requests serially, since we know that when the server finishes responding to our nth request, it should be ready to handle the (n+1)th request. One way to accomplish this is with a big long chain of .thens, as by the time we're in a .then, we're guaranteed that its preceding promise is completed. And one way we can construct that chain is with a reduce:

urls.reduce(
    (acc, url) => acc.then(() => rp.get(url).then(sendRelatedRequests)),
    Promise.resolve()
);

[].reduce takes two arguments: the reducing function, and the initial value. We need to start with a Promise, because our reduce function assumes that the accumulator acc has a .then on it.

For the reducing function, we have an accumular, and a url. Each time, acc is the existing serial chain of promises, and we add another .then on to it. The important part is that the function in the .then handler is not immediately creating the promise, because that would mean the request is immediately sent. Instead, passing the function expression means the Promise isn't created until the .then handler is invoked, and since the .then handler is invoked until its preceding Promise is complete, we get our serial behavior.

Also, since the requests don't care about each other, we don't need to use the arguments that are coming from the previous promise, so the function expression doesn't use its arguments.

The one last catch (hoho) is that if any of the rp.get(url) promises fail, then all of the subsequent .thens are skipped, as the promise flow dictates that it should jump to a .catch handler, if one exists. So, to guarantee that we do make all the requests that we wanted to, we need to add a catch handler to each of the promises in the chain.

urls.reduce(
    (acc, url) => acc.then(() => rp.get(url)
                           .then(sendRelatedRequests)
                           .catch(console.error)),
    Promise.resolve()
);

Connecting to Mongo with a self signed CA on a JVM in Kubernetes

2017-12-10T00:00:00Z

At $WORK, we're creating an internal platform on top of Kubernetes for developers to deploy their apps. Our Ops people have graciously provided us with Mongo clusters that all use certificates signed by a self-signed certificate authority. So, all our clients need to know about the self-signed CA in order to connect to Mongo. For Node or Python, it's possible to pass the self-signed CA file in the code running in the application.

But, things are a little more complicated for Java or Scala apps, because configuration of certificate authorities is done at the JVM level, not at the code level. And for an extra level of fun, we want to do it in Kubernetes, transparently to our developers, so they don't have to worry about it on their own.

[[MORE]]

err, wha? telling the JVM about our CA

First off, we had to figure out how to tell the JVM to use our CA. And luckily since all the JVM languages use the same JVM, it's the same steps for Scala, or Clojure, or whatever other JVM language you prefer. The native MongoDB Java driver docs tell us exactly what we need to do: use keytool to import the cert into a keystore that the JVM wants, and then use system properties to tell the JVM to use that keystore. The keytool command in the docs is:

$ keytool -importcert -trustcacerts -file <path to certificate authority file> \
  -keystore <path to trust store> -storepass <password>

The path to the existing keystore that the JVM uses by default is $JAVA_HOME/jre/lib/security/cacerts, and its default password is changeit. So if you wanted to add your self signed CA to the existing keystore, it'd be something like

$ keytool -importcert -trustcacerts -file ssca.cer \
  -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit

(Even this very first step had complications. Our self signed CA was a Version 1 cert with v3 extensions, and while no other language cared, keytool refused to create a keystore with it. We ended up having to create a new self-signed CA with the appropriate version. Some lucky googling led us to that conclusion, but of particular use was using openssl to examine the CA and check its versions and extensions:)

$ openssl x509 -in ssca.cer -text -noout
// Certificate:
//     Data:
//         Version: 3 (0x2)
//         Serial Number: ...
//         ...
//         X509v3 extensions:
//             X509v3 Subject Key Identifier: ...
//             X509v3 Key Usage: ...
//             X509v3 Basic Constraints: ...

Another useful command was examining the keystore before and after we imported our self signed CA:

$ keytool -list -keystore /path/to/keystore/file

as you can look for your self-signed CA in there to see if you ran the command correctly.

Anyway, once you've created a keystore for the JVM, the next step is to set the appropriate system properties, again as out lined in the docs:

$ java \
  -Djavax.net.ssl.trustStore=/path/to/cacerts \
  -Djavax.net.ssl.trustStorePassword=changeit \
  -jar whatever.jar

Since the default password is changeit, you may want to change it... but if you don't change it, you wouldn't have to specify the trustStorePassword system property.

handling this in kubernetes

The above steps aren't too complicated on their own. We just need to make sure we add our CA to the existing ones, and point the JVM towards our new and improved file. But, since we'll eventually need to rotate the self-signed CA, we can't just run keytool once and copy it everywhere. So, an initContainer it is! keytool is a java utility, and it's handily available on the openjdk:8u121-alpine image, which means we can make a initContainer that runs keytool for us dynamically, as part of our Deployment.

Since seeing the entire manifest at once doesn't necessarily make it easy to see what's going on, I'm going to show the key bits piece by piece. All of the following chunks of yaml belong to in the spec.template.spec object of a Deployment or Statefulset.

spec:
  template:
    spec:
      volumes:
      - name: truststore
        emptyDir: {}
      - name: self-signed-ca
        secret:
          secretName: self-signed-ca

So, first things first, volumes: an empty volume called truststore which we'll put our new and improved keystore-with-our-ssca. Also, we'll need a volume for the self-signed CA itself. Our Ops provided it for us in a secret with a key ca.crt, but you can get it into your containers any way you want.

$ kubectl get secret self-signed-ca -o yaml --export
apiVersion: v1
data:
  ca.crt: ...
kind: Secret
metadata:
  name: self-signed-ca
type: Opaque

With the volumes in place, we need to set up init containers to do our keytool work. I assume (not actually sure) that we need to add our self-signed CA to the existing CAs, so we use one initContainer to copy the existing default cacerts file into our truststore volume, and another initContainer to run the keytool command. It's totally fine to combine these into one container, but I didn't feel like making a custom docker image with a shell script or having a super long command line. So:

spec:
  template:
    spec:
      initContainers:
      - name: copy
        image: openjdk:8u121-alpine
        command: [ cp,
                   /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts,
                   /ssca/truststore/cacerts ]
        volumeMounts:
        - name: truststore
          mountPath: /ssca/truststore

      - name: import
        image: openjdk:8u121-alpine
        command: [ keytool, -importcert, -v, -noprompt, -trustcacerts,
                   -file, /ssca/ca/ca.crt, -keystore, /ssca/truststore/cacerts,
                   -storepass, changeit ]
        volumeMounts:
        - name: truststore
          mountPath: /ssca/truststore
        - name: self-signed-ca
          mountPath: /ssca/ca

Mount the truststore volume in the copy initContainer, grab the file cacerts file, and put it in our truststore volume. Note that while we'd like to use $JAVA_HOME in the copy initContainer, I couldn't figure out how to use environment variables in the command. Also, since we're using a tagged docker image, there is a pretty good guarantee that the filepath shouldn't change underneath us, even though it's hardcoded.

Next, the import step! We need to mount the self-signed CA into this container as well. Run the keytool command as described above, referencing our copied cacerts file in our truststore volume and passing in our ssCA.

Two things to note here: the -noprompt argument to keytool is mandatory, or else keytool will prompt for interaction, but of course the initContainer isn't running in a shell for someone to hit yes in. Also, the mountPaths for these volumes should be separate folders! I know Kubernetes is happy to overwrite existing directories when a volume mountPath clashes with a directory on the image, and since we have different data in our volumes, they can't be in the same directory. (...probably, I didn't actually check)

The final step is telling the JVM where our new and improved trust store is. My first idea was just to add args to the manifest and set the system property in there, but if the Dockerfile ENTRYPOINT is something like

java -jar whatever.jar

then we'd get a command like

java -jar whatever.jar -Djavax.net.ssl.trustStore=...

which would pass the option to the jar instead of setting a system property. Plus, that wouldn't work at all if the ENTRYPOINT was a shell script or something that wasn't expecting arguments.

After some searching, StackOverflow taught us about the JAVA_OPTS and JAVA_TOOL_OPTIONS environment variables. We can append our trustStore to the existing value of these env vars, and we'd be good to go!

spec:
  template:
    spec:
      containers:
      - image: your-app-image
        env:
          # make sure not to overwrite this when composing the yaml
        - name: JAVA_OPTS
          value: -Djavax.net.ssl.trustStore=/ssca/truststore/cacerts
        volumeMounts:
        - name: truststore
          mountPath: /ssca/truststore

In our app that we use to construct the manifests, we check if the developer is already trying to set JAVA_OPTS to something, and make sure that we append to the existing value instead of overwriting it.

a conclusion of sorts

Uh, so that got kind of long, but the overall idea is more or less straightforward. Add our self-signed CA to the existing cacerts file, and tell the JVM to use it as the truststore. (Note that it's the trustStore option you want, not the keyStore!). The entire Deployment manifest all together is also available, if that sounds useful...

Uninstalling Mersive's Solstice Audio Driver from MacOS Mojave

2019-08-06T00:00:00Z

This was a minor annoyance, but since my Google searches were particularly unfruitful, I figured I'd make a note of it. Our WeWork uses the Mersive Solstice client for broadcasting our devices to the meeting room televisions. It has the option to install its own audio driver to try to mirror the audio on its own.

The client's built-in audio driver is called "Desktop Audio Streaming Device" and it shows up as an output and an input device. It also completely failed to work at all. The only way I managed to get sound on to the TV was by using AirPlay after connecting with the Solstice client - basically using it as little as possible.

Uninstalling audio drivers is apparently something weird in OS X - the device was listed in the Audio Midi Setup built-in macOS app, but couldn't be uninstalled from the app. Some wild googling directed me to /Library/Audio/Plug-Ins/HAL/, or /Library/Preferences, or their ~/Library counterparts. In the end, it was a kext in /System/Library/Extensions: SolsticeAudioDevice.kext. Trashing that file and restarting the computer fixed the issue.

Getting started with standalone Kayenta + Referee for Automated Canary Analysis

2021-04-16T00:00:00Z

Automated canary analysis is a deploy strategy wherein you direct a subset of your (production?) traffic to a canary version of your application. The idea is that your baseline application and your canary application will then be shipping metrics off to your metrics provider, and Kayenta + Referee help you determine whether your new code fails any metrics! This replaces developers manually looking at dashboards & logs during deploys.

Kayenta is an open source Java microservice for performing automated canary analysis - it's just one microservice of many that make up Netflix's Spinnaker platform. Referee is an open-source React frontend that sits in front of Kayenta and allows for rapid iteration of your canary configurations. This blog post will outline the steps I took to get this set up at $WORK recently!

You can read through the details below in this blog post, or go straight to the kayenta-referee-quickstart repo to get started right away.

[[MORE]]

Configuring Kayenta

Starting up Kayenta consists mainly of configuring its three data stores in the kayenta.yml config: METRICS_STORE, CONFIGURATION_STORE, and OBJECT_STORE.

For the METRICS_STORE, choose the one that your applications is shipping its metrics to. Kayenta supports Atlas, Google, Datadog, graphite, New Relic, Prometheus, SignalFX, and Wavefront. You'll need to obtain the appropriate API keys to allow Kayenta to query the metrics API, which will depend on the datastore you're using. For datadog, the config excerpt looks like:

  datadog:
    enabled: false
    metadataCachingIntervalMS: 900000
    accounts:
      - name: my-datadog-account
        apiKey: xxxx
        applicationKey: xxxx
        supportedTypes:
          - METRICS_STORE
        endpoint.baseUrl: https://app.datadoghq.com

For CONFIGURATION_STORE and OBJECT_STORE, you can start out with the built-in in-memory-store, and this is already configured in the quickstart repo. Later, you can switch that to S3/GCP/Azure if you need to persist your configs/objects.

  memory:
    enabled: true
    accounts:
    - name: in-memory-store
      supportedTypes:
      - OBJECT_STORE
      - CONFIGURATION_STORE

running kayenta + referee

Over in the kayenta-referee-quickstart Github repo, there are dockerfiles and a docker-compose.yml that you can use to run both of the microservices locally. Fill out your metrics store API keys in the kayenta.yml config at the root of the repo, and then you can start it up:

docker-compose up -d

The Kayenta API documentation will be available at http://localhost:3001/swagger-ui.html, and Referee will be running at http://localhost:8090/dashboard. If the containers don't come up, check their logs for any telling messages:

docker-compose logs kayenta
docker-compose logs referee

If you decide you want to deploy Kayenta & Referee into your internal infrastructure, you'll need to provide a Redis instance for Kayenta to talk to. This will depend on your internal infrastructure, but in a pinch, you could use a Docker container running Redis even without a persistent data volume backing it, if you don't need to store the Canary configurations inside Kayenta.

integrating kayenta + referee into your production deploys

If you're embarking on the canary production deploy integration road, one of the most useful tips I came across was to start out with an asynchronous canary pipeline separate from your existing production pipeline. Have the canary pipeline run in parallel to your existing production deploy, and don't let the canary pipeline block or even fail deploys. Instead, the idea is to set up the canary pipeline with a few metrics and have it gather data on every production deploy until you can see whether the metrics you chose are appropriate and would catch real production issues.

canary configurations

Configuring a canary consists of constructing the POST body for the standalone canary endpoint; there's probably enough material there for its own blog post, so we'll defer that to a potential part two!