Mad Magazine was how we learned the truth behind the curtain. The Cold War was on and Spy vs. Spy was as good as Boris and Natasha. Product commercial parodies taught us how to think through the hype. For those of us of a certain age “what me worry” is a funny tag line, but it’s not a systems contracting methodology.
Contracting for a system implementation and integration requires understanding how systems are built, structuring the contracts accordingly to avoid overcharges and place responsibility where it needs to be at each key project gate. From what I’ve been able to glean from the papers and news, it seems the healthcare.com contractors were brilliant in contracting. What did they do, and as buyers how can we not repeat those practices to our detriment?
1 – Final authority was unclear: each ship can have multiple Captains, but only 1 Master of the Vessel
Who is running the project? You or the contractor? This problem was solved eons ago during the ancient development of maritime trade. To this day, each ship has a crew, and can have more than 1 Captain, but only 1 Master of the Vessel. Make sure yours is highly qualified and empowered to demand answers to key questions and not be afraid of having a contractor perform an end-run to another executive or have their jobs put on the line. I learned this the hard way when I competed against Big Blue while they were at the height of their power. Their form of an end-run was taking a hesitant CIO to their boss and have this CIO explain, in front of Big Blue, why the CIO didn’t agree with their particular proposal or recommendation. After being provoked, the CIO would lose their cool, and then the Big Blue rep would respond “unfortunately Bob/Jane is not objective in this matter”. This probably contributed to the saying ‘CIO means Career Is Over’. I knew several instances where this vendor had an office near their customer’s CIO, and just as big. Think the CIO was going to question anything? That’s all changed now, but contractors, especially Gov’t contractors, are sharp elbowed to this day.
Our recommendations: Assign a ‘Master of the Project’, which is more than the lead Project Leader, since most PMO’s are staff oriented without line authority. Having a ‘Master of the Project’ does not mean everyone else if off the testing hook. It simply means the Master has the final say on acceptance, payments and Organization Change Management, and it is well understood any end-run around them would not be painless for the contractor. Just as the Master of the Vessel has an Executive Officer to carry out orders at a tactical level, have each contractor assign an empowered (technical, not sales) tactical leader, reporting directly to your Master. The Master of the Project should also have the authority to limit the number of participants to ensure focus. Look at the org chart. When you think you may have too many participants to control, remember this astronaut quote, “how do you think you’d feel if you knew you were on top of two million parts each built by the lowest bidder in a government contract?”
We also recommend establishing a new testing stage, Pre-UAT. Contractually, User Acceptance Testing (UAT) typically triggers a payment event and the beginning of the warranty period. We recommend scheduling a Pre-UAT step as a way to thoroughly test the system as if it was in final acceptance phase, but have the contractor repair any defects identified at this late point at their own expense. Once the system is tested bullet-proof, it can then be promoted to the payment triggering final User Acceptance Testing phase. Build the Operating Model into the contract, in detail.
2 – No one specified ‘who tests what?’
Systems have 3 levels of testing: developers run Unit Testing (UT) to confirm the code works, they then perform Systems Integration Testing (SIT), to ensure everything talks to everything as required to go end to end, and finally, they present the finished product to the customer for User Acceptance Testing (UAT). If the contractors were “not responsible” for testing, does this mean they did near-nothing? Having end users, Beta Testers or otherwise, banging away to test the site is not Unit Testing or Systems Integration testing, it’s hardly User Acceptance Testing – I call end user testing that late in a project ‘Panic Testing Before Updating LinkedIn Profile’. If these contractors did not conduct testing, it implies they did not have a robust testing environment set up, which means they would have trouble mapping any end user found defects to the 500 million line codebase. Repairs would be expensive, but at cost+, profitable.
Our recommendations: Require, contractually, all code be tested at the Unit and Systems Integration levels by the contractor, followed by Pre-UAT (see above), and only then at the per-phase User Acceptance level to trigger the payment event. Include end to end and stress (volume) testing, as well as resource efficiency at extreme loads and handling outlier error scenarios. We also recommend using an outside Static Code Analysis service/tool to ensure each module has a coherent architecture and is not, in developer vernacular, a ‘Big Ball of Mud’, which inherently is unstable and difficult to enhance and maintain.
3 – They locked in profits through the cost+ method.
This probably answers why they kept coding even though they didn’t know the full range of specifications. It also explains why they kept coding, writing 500 million lines of code, or 10X the size of an operating system, one of the most complex yet optimized pieces of code around. I don’t know this for sure, but I’ll bet they also used a blended rate for consultant hourly fees, allowing them to substitute the cheapest resources, which would also take longest, and maximize profits. It also stands to reason they probably could charge full freight for a copy and paste of previously written code (which may also explain how they hit the 500 million lines of code threshold). It encourages sloppiness, which can affect system stability and performance. The standard answer of “we can always throw more Cloud resources on it” is true from a Computer Science sense, but less so for the company paying for those added resources for the next 10 years.
Our recommendations: Fixed fees are always better, and a means to determine these fees is to run a ‘T&M with a cap’ scoping project up front, followed by a comprehensive project plan and cost forecast tied to the scoping efforts results. We also recommend having the project phased, where each phase has a distinct price. In those cases where a contractor will not agree to a fixed price (and we can see where that can be the case), we recommend each phase is billed as T&M with a cap on total spending for the project and phase.
Assuming the vendor has done your type of project before, they should have a ‘starter-kit’ of reusable, already optimized, templates and routines, typically applied with some modifications per project. During negotiations, have them describe their starter-kit, license it as part of your contract, and determine how it will reduce the amount of new code being written. In one healthcare.gov contractor’s case, they seem to be involved in several State Exchange websites, implying repurposing of a baseline starter-kit. Be hesitant of someone who does not have either recent hands-on experience or a starter-kit with clean IP ownership. At the least, protect yourself contractually through the acceptance criteria process and associated payment terms.
Bottom line, systems development and integration is hard, with different technologies and contracting terms, all of which requires serious and relentless focus, by someone with ultimate authority. You need a working, viable system; they need to turn a reasonable profit on the engagement. For both parties it’s about execution and the right contract terms will set the tone and steps to make that happen.
BTW, Happy 60th Birthday, Mad Magazine. Here’s Mad’s current editor talking to CNN about the magazine’s history. http://www.cnn.com/video/data/2.0/video/showbiz/2012/11/02/natpkg-mad-magazine-60th-anniversary.cnn.html. Thanks for the education, Alfred.
Richard Eichen is the Founder and Managing Principal of Return on Efficiency, LLC, http://www.growroe.com , focusing on companies, initiatives and products where technology is the primary means of delivery and revenue. He is one of their senior Turnaround, Transformation, Program Rescue and Process Rescue leaders. As a Change Agent, Trusted Advisor, Program Leader and Interim Executive, Rich has over 25 years hands-on experience reshaping companies, Operations, IT/Systems Integration and strategic initiatives. He can be reached at email@example.com, and followed on Twitter, @RDEgrowroe