Saturday, February 5, 2011

How to completely fail at BDD

Are you interested in introducing BDD to your team? Don’t try and do it like this under these circumstances. Learn from my failure.

Having experienced ambiguous requirements specifications and inadequate testing of the applications I’ve worked on during my career as a software developer, when I saw BDD I thought I saw the future. In particular the Gherkin based implementations (Cucumber, Specflow) with their Given/When/Then descriptions of scenarios for product features that could lead to executable specifications blew me away as full of potential.

I was incredibly eager to try this stuff out on a real project.

I played around a bit in my spare time with Cucumber. I learned how features were specified in plain text feature files and tests related to them were implemented in step definitions. You match the scenarios you describe in your feature files to step definitions that exercise your application code using regular expressions. I’d never really used regex that much so was initially a bit concerned that this would be a problem. It turned out though that with a small grab bag of regex tricks you could do most things you needed.

One of my first real problems I ran into was that I was looking to try BDD on an upcoming C# project. That put me in a minority group. I don’t entirely pretend to understand why, but much of the C# ecosystem seems pretty closed off, lacking in the use and acceptance of rich open source projects that I was used to in my prior life working on projects on the Java stack. (As an aside, I even notice differences when I interview C# programmers compared to Java programmers. The way they think. The things they have and haven’t read. They typically haven’t done unit testing, typically haven’t used any open source software. In general, their view of the software development world is quite shuttered. I know this is a gross generalization, but it’s noticeable. It’s a shame, because C# seems like a pretty cool language…)

I quickly found Cuke4Nuke which allows you to implement your step definitions in C# rather than Ruby (the norm for Cucumber) and Cuke4VS which would give you syntax highlighting etc. when editing feature files in Visual Studio. Here I encountered my second couple of problems: Cuke4Nuke wasn’t ready for .NET 4 and Cuke4VS even to this day doesn’t work on VS2010. Of course our project was using .NET4 and VS2010. Fixing Cuke4Nuke wasn’t too hard, I fumbled my way through that OK. Fixing Cuke4VS though was beyond me. Well, beyond the amount of effort I wanted to put in anyway. I’ve not done much with C# and nothing with Visual Studio addins like this. I sorta got it working, but nothing I could push back as a quality patch to the Cuke4VS project.

Then I discovered Specflow.

Specflow has some distinct goodness:
  • It’s Gherkin compatible, which means you describe your requirements in terms of features and corresponding given/when/then scenarios. While I lack the experience that would come from doing substantial real work in this, it intuitively feels “better” to me than the table based approach taken by Fit, Concordion etc.
  • It’s good to go with VS2010 and .NET4.
  • No Ruby required. With Cucumber, even though you can write your step definitions in C# courtesy of Cuke4Nuke, you still need a Ruby installation to parse your .feature files and call out over the wire to your C# steps. I’m not the first to observe that the .NET/C# community is kind of reluctant to install “weird shit” like Ruby that “doesn’t come from Microsoft”.
Sadly, at this point in time it also has some challenges:
  • Documentation. The main problem is the documentation, or lack thereof. Pull down the “Specflow guide” PDF and you’ll find a clear work in progress, incomplete and missing sections, editing notes etc.
  • Examples. Github hosts a SpecFlow-Examples project, but it needs some TLC. There are some reasonable examples in there, but quite where is not immediately obvious. All the action is inside ASP.NET-MVC/BookShop, not the features folder. Github is also potentially another barrier to entry for people. I don’t know how many C# developers are familiar with Github, but none on my team are.
  • VS silliness. Not Specflow’s fault, but I had hoped to use Visual C# Express for the QA people on my team to create .feature files. Sadly Microsoft has seen fit to not support Add-ins so it doesn’t work. Another barrier to adoption, at least for my team.
  • Stability. I never actually experienced this myself, but two of the engineers on my team said they had stability problems that they attributed directly to Specflow. It “mangled the project file.” Things “wouldn’t compile and it was complaining about Specflow stuff”. I never really got great details on this, so it’s hard to tell how much if any was directly down to Specflow. How much was me with rose-tinted glasses seeing no problem attributable to Specflow and perhaps others less interested in a new technique seeing every problem attributable to Specflow.
In general, you’re going to need a lot of patience, a genuine desire in a majority of the team to do something quite radically different to “normal” and you’ll need to draw a lot of information from the Cucumber community and then be comfortable using the mailing list for additional support. I’m actually OK with this, but it can be a real barrier to entry for a lot of people.

There were also some serious non-technical challenges; people skills and organization/political in nature.

Process and regulatory barriers.
My team makes software that is subject to 21 CFR part 11 regulations. That’s stuff that tries to help make sure any pharmaceutical drugs you take are safe. That kind of industry cascades regulations through to any software related to clinical studies to make sure the software is of high quality. Our operation can be (and frequently is) audited by our clients to make sure we comply and can (though rarely is) audited by the FDA. Having devised a “winning formula” that gets us successfully through audits people are naturally reluctant to make changes in anything that would get seen by an auditor. Having your “requirements” in .feature files alongside source code then is a serious brown trouser moment for some people.
  • “They could end up seeing our source code!” Yes, believe it or not, a process designed to make sure the software involved in clinical trials doesn’t actually often have people look at the source code. They prefer Word documents, signed in duplicate (or more).
  • “We need to show them a traditional requirements specification with numbered requirements that we can trace through to test cases.” Because that’s the only possible way to do this. Of course.
  • “We have to review and sign off the test scripts before they’re executed, and prove that we have done so.” I can see how this made sense 20 years ago when test scripts where a documented set of steps a manual tester followed. But what does this even mean when your “test script” is a plain text .feature file and a corresponding set of step definitions in C#?
I actually have to commend the internal quality expert I spoke to for being very receptive to new ideas and trusting my assertion that I could probably generate a traditional style requirements specification and traceability matrix that mapped features to test cases from .features files. She was less resistant than some people on the team.

Skills Deficits
The people on the team responsible for testing the product and generating the nice bundle of validation paperwork those who audit us like to see primarily have a background in testing software manually. I’d looked over their manual test scripts before (long Word documents with lots of boilerplate and then long rambling detailed steps with expected outcomes and places to mark discrepancies) and got the impression they could be better. They seemed to meander quite a lot, with a lack of clarity over exactly what precisely one was testing. I do now know how anyone could review them and spot test cases that were missing. I’d even taken a manual test script for a fairly trivial feature that ran to thirteen pages and felt pretty smug about being able to reduce it down to one short .feature file.

Some people on the team had asked me whether I thought those folks could use Specflow. Could they write good .feature files and the corresponding step definition classes? It was a fair question, and I was pretty sure some might struggle. That said, we already had a test automation tool called Test Partner that involved folks writing things in VBA, so if they could do that, could this be much harder? And I didn’t see this as a reason to indicate that Specflow and BDD was wrong. If BDD is a good idea it’s a good idea. Sure, maybe we needed to train folks or modify the composition of our team, but that was all.

But oh did I underestimate just how much they would struggle. At this point in time I wasn’t sure if Visual Studio Express would work with Specflow. So I suggested they just try. There then followed quite the flurry of emails. People who ostensibly test things for a living were unable to reliably determine whether they could use Visual Studio Express or a full featured version.

And that was just the installation.

Then we got into writing a few .feature files. The first one wasn’t…very good. This was a little disappointing as I’d circulated a number of links to articles on the web about writing “good” features but, OK, it was a first attempt. A couple of the software engineers very graciously (much more gracious than I’m being right now, but I’m currently enjoying my second Left Hand Brewing Company’s Fade to Black) pointed out some things they might do to improve them. Sadly subsequent attempts at writing .feature files were no better. It was as though the advice offered was ignored or not understood.

We didn’t even get near them implementing a step definition. I really have to wonder what they’ll be putting together in Test Partner in those VBA scripts it uses. Thank goodness we have unit tests.

Incumbent Test Framework
As indicated above, we have available (actually we’re pretty much obliged to use) a product called Test Partner. Of course it’s only for QA people, especially given the per-seat cost. It’s all kinds of horrible so far as I can tell after looking at it. It doesn’t even make it easy to place your test scripts under source code control, preferring instead to keep them inside a database (but of course…) and though you can export them (base 64 encoded, naturally) it’s hardly ideal.

In addition to the original Test Partner investment, several people have spent a not inconsiderable amount of time and money building an additional layer (or framework) on top with functionality for such things as logging and managing collections (*cough* reinventing the wheel *cough*).

Of course the political reality attached to this state of affairs is that any other pretender to the crown has got its work cut out.

Cultural suitability
My love affair with computers started back in the early 80s around age 10. A friend had an Apple IIe computer and we played Bouncing Kamungas and other mad games, we entered epically long listings that sometimes worked. The school had a Commodore PET. I learned to program simple games in BASIC. I got a Vic 20 and wrote more games, even sold some to people at school. Then later I got an Amstrad CPC 6128 complete with it's wacky 3" disk drive and after that an Amiga. I worked in an independent computer store and wrote code for their business customers at age 16. I went to college and learned more. I got a “proper” job. It’s waxed and waned ever since. There’s still nothing quite like the fun of programming I think. But the reality of a corporate job can take the shine off that – it’s not the same as sitting there with your best buddy late at night debugging a game you’ve spent all day writing. So throughout my professional career my enthusiasm has varied along with different projects, languages, technologies, jobs and bosses.

Nevertheless, several years back I took a leadership position, got interested in agile and discovered a new found passion for reading about and playing with new technologies and approaches to building software. I joined twitter, found a community of thought leaders and everyday practitioners, started reading books and blog posts on all of this, got excited, started a transition at work. Mostly successful. But not successful enough I guess. I’m still one of the very few people that does that kind of thing. And when you’re perhaps the only one on a team that reads and writes blogs, uses twitter, reads books, wants to try doing things differently it can be tough. You're seeing so much that other people aren't.

The fact is I’ve been having to sell BDD to almost everyone. Maybe one person really “gets” the potential. A few more are fairly excited about it. But it's so far outside their comfort zone.

When I first managed to get some consensus that we should try this I hadn’t done enough with Specflow myself, almost all my experimentation up to this point was with Cucumber. I wasn’t worried though, the guy that was going to try it was an excellent engineer. I was more than a little surprised then when almost every hour a new “issue” was presented on why this couldn’t possibly work. I hadn’t seen that coming.

Culturally, my current team just isn’t ready or interested in something like this. And I have to accept that I can’t necessarily change that. Maybe I should have done a lot more work with Specflow myself first and had good examples to show people. I'm not sure.

Blah, blah whine! WTF am I gonna do about it?
I don’t know yet. Not for sure. I’m thinking maybe I can contribute to the Specflow project and help out with the documentation. This isn’t entirely appealing though if we won’t even be using it. So maybe I’ll use it on the sly, even if we’re not “officially” using it for our project maybe I can keep trying it out. And use that experience to develop something like the documentation that people on my team felt needed to be there.

If you have any suggestions, let me know...

OK, self-indulgent rant over.