|By Jesse Randall Warden||
|April 15, 2009 01:00 PM EDT||
Five weeks ago, I started a project with Enablus, a firm that works with mainly startups to build products. I loathe service work, and love product work, so am really glad to be working with them. Even better, they are only 30 minutes from my house. I don’t really need to go more than 2 to 3 times a week, but I like how close they are when I do. One thing that struck me in the initial project discussions was that Enablus was really big on process and lingo. Darrell Ross, the project manager and partner I’m working with, although hard to read initially, still showed signs of uneasiness when I didn’t give him the answers he was wanting to hear in the initial discussions. For example:
“Do you utilize Continuous Integration?”
“Uh… what the hell is that?”
“You know, everyone on the team checking in code into a source control system every day, utilizing Test Driven Development practices, automating your builds, etc.”
We’d constantly go through this. He’d ask a serious question like, “Do you utilize OOP and an software framework(s)?” and I’d respond with some cynical response questioning the merits of when OOP and frameworks are applied, or what I did and did not like about a particular framework, and how I don’t always officially follow their implementations… never really answering his question and thus not giving him confidence. I was constantly amazed at how structured Darrell made things sound, and he probably wondering how the hell a positive impression of “Jesse Warden” was formed in his head in the first place based on the answers I was giving him.
In all instances, we’d discuss the details of my responses, and he seemed to get the details he needed, feel better, and move on. Too ease his stress, I opened up and said that I’ve never really worked in places that were above a Level 3, and even those I felt never followed the processes he was mentioning, specifically Waterfall or Agile development, to the letter. Later in my consulting and contracting career, I had moved to using some concepts of Agile such as iterative development, and having builds of the software regularly on a schedule (usually weekly) even if the client didn’t need one for weeks or months. It had nothing but positive ramifications in the long run, so I stuck with what worked and saw other successful people doing similar things. It’s really a simple process (my version): Have a build working by Friday, every Friday.
I’m not sure Darrell believed me that everywhere I’ve worked, whether full-time or consulting/contracting, did not use a standardize process such as Agile. Maybe it did and I just wasn’t privy to the details? Again, I never cared till this year. Either way, nothing sounded as strict and defined as Darrell made it sound as he described the Agile process to me. Hence the point of this series of posts.
I’ve never been interested software development processes. To me, it’s always been more important to be really good at the 3 main things that count: knowing your platform, knowing your tools, and knowing the language you code in. The better you know those 3 things, the more capable you’ll be at developing software, helping those on your team develop software, and thus ensure your team will be successful. As Dave Wolfe has said, it takes a village; a great team can ensure success. Whether the project actually succeeds isn’t really up to the software team at that point; hence the second rule to the “8 of 10 software projects fail” is “9 out of 10 are perceived as failures”. Even if you create an awesome piece of software, management, stakeholders, and/or clients can still perceive it as failing.
As I’ve found throughout my career, there have been more pressing issues than “process”, such as:
- getting the dang design comps
- making a workflow where fonts will compile on both machines (Mac and PC)
- coming to a consnsus with team members on how the framework you are using should be implemented
- finding out what the heck we are building really is
- finding out how the heck what we are building really works
- resolving source control issues
There are tons of other things, but each has the capacity to reach fire drill status, and thus become a major time sink and productivity killer. As you know, losing time is the LAST thing you want to have happen in software development.
It also depends on the type of work. Working with Design Agencies, there was no process. Not because Agencies are dumb, but rather because the project timelines Agencies have are typically significantly shorter than traditional software or product development. On average, 2 weeks vs. 4 months. While a deadline can stretch to 4 months, you only work 10 man days on it, and of those 10, 5 are done initially, and the other 5 are done over the course of 2 or 3 months on client changes. In short, if you have 5 days to complete a complicated coding project, you’re more focused on getting things done than you are worrying about encapsulation, continuous integration, and frameworks. Writing code on day 1, and wondering who in the heck has time for UML and test cases. Agencies are deadline driven, whereas in software development, nothing is ever on time.
In consulting, you’re typically working within the constraints of your client’s company. It can go down two different ways. Either you get treated like a service shop, and requirements are thrown over the wall, and you complete how you see fit. Otherwise, you integrate with an existing team, and work within their processes, trying to improve them where you can. Contracting is usually the same way, although, you usually have little to no say in the process.
Even including regular work (W2 at a software shop), all instances can significantly have their chances of success improved through skill. If you’re skillful in the aforementioned 3 areas (platform, tools, language), it doesn’t matter how screwed up things are; you can always depend on skill to make or break a project.
…or so I thought. Going back to the 2nd rule of software development, “9 out of 10 software projects are perceived as failures”, I started to have more of the second rule happening vs. the first. In those projects, more of my time would be spent doing non-software things, like pitching a tent in the project manager’s office, discussing details with stakeholders, drilling sales on what they were really trying to accomplish, etc. In the end, trying to define success to ensure my team and I weren’t setup to fail from the beginning. Being setup to fail, yet having a team with mad skill, results in you months later outside the office smiling on your umpteenth beer regaling all in the pimp architecture you created that never did, nor will, see the light of day. Creating awesomeness with no result is extremely more frustrating than flat out failing. The positive to failing is you can learn from it. Failing, even though technically you succeeded, is just straight ridiculous, and it was at those points I started to become more interested in the software development process. I realized I needed to find a team who knew all these processes I’d read about on the internet blogs.
Enablus is one of those teams. They clearly know Agile, as far as I know it 5 weeks in.
I got Darrell to explain how Agile, specifically the Scrum part, works. This is my paraphrased, and re-interpreted version of our discussions, and my working with the process for a few weeks. If you have a good idea, do market research to ensure it’ll sell, then you just need a good team to execute it. The execution on the software side for product development utilizes an Agile approach.
The Agile approach, for the engineer, starts by defining “User Stories”. User stories are a set of sentences that describe what the user experiences while using the software. These can be derived from the wireframes; design comps aren’t a requirement. A user story can be expressed like so:
Login: User inputs Username and Password and clicks “Start Application”. Login is validated. For all other login instances, the Dashboard is displayed.
Notice there is no implied technology or design construction details. The user story merely describes what the user see’s, what interactions the do, and what the results of those interactions are. That same user story could apply to Flash, Silverlight, AJAX, PHP, etc. It’s technology agnostic; the point here is to document a user experience to ensure we’ve captured it well enough to be understood, and thus executed upon. The same goes for design. It doesn’t say “2 text input fields for username and password, with bold headings to the left, aligned center”. It’s left up to the interpretation of the designer to execute that user story, and hopefully provide direction for the GUI programmer through design comps assuming the user stories were derived from the wireframes, and not the design comps.
Notice also, it’s not a “feature”. It doesn’t say, “The user can login”. That leaves things massively open to interpretation, and solves no business value even if the feature is “completed”. As we all know, not all logins are the same. Documenting features, and executing them is one of the number one ways to be setup to fail, and is also one the main reasons Waterfall doesn’t work. If a software spec says, “The user must be able to log into the system”, but doesn’t describe registration, single sign-on, error scenarios, and how it’s supposed to look, you’ll end up with software that the client doesn’t like.
Some user stories are extremely high level. These are sometimes called meta-stories because they actually encompass a wide array of smaller user stories. Describing a login scenario can have a multitude of smaller user stories within it such as “user is presented with an error dialogue stating that they must first activate their account via the confirmation email before gaining access to the entire application’s feature set”. The level of detail only matters in so much as to make sense to the designer and engineer executing it. If a user story encompasses weeks of coding, it’s probably a meta-story and should be broken down into more manageable user stories that are easily defined, and more easily tackled in a reasonable time frame. What’s reasonable and doable is different for each team.
You create user stories for your entire application. What does, rather what SHOULD, the user be able to see, do, and experience?
These user stories go into a “Backlog”. This backlog is a list of all the user stories, say 50… or a hundred. How many user stories you have are irrelevant. What’s important is that they are all documented in one place called the Backlog. You need to ensure all of the user stories are valid with the stake holders. It also helps to go over them with your entire team to ensure they make sense to everyone involved.
As you move through your project, you may find you’ll need more user stories to account for things you didn’t think about, or for additions/modifications to existing user stories. You then just add them to the back log. It’s normal for a backlog to grow during a project life cycle. However, you do not have to complete the entire backlog to complete the project. A lot of times, the stakeholders will make a business decision that 40 of the 60 user stories is enough to have a great product, and say “ship it”. That’s one of the key benefits to Agile is that you have working software throughout the project, and at anytime can make the determination of when it’s “done”… or is just a version 1.
Once your done with your backlog, you then have your team go through and assign points to each user story. To do that, you need a Point System. A point system is defining a metric for “how challenging something is”. In simple terms, a 1 is easy and a 5 is hard. It doesn’t have to be 1 through 5; it could be 1 through 10 or whatever. It’s best if they are numeric values because you use the points to measure a variety of statistics about your project. Additionally, each team may have their own definitions of what is challenging. While it’s best to use the same 1-whatever metric for the client and server team, the server team will most likely think creating a new server-side method a “1-easy” whereas a client developer a “5-hard”.
Once you’ve defined your point system and gained consensus amongst the team, you then go and assign a point value to each user story. This in essense gives each user story a “value”; how many points a user story worth. If you have multiple team members working on different parts of the system, say server-side and client side, you then have a point value added to each, such as “2 - mostly easy” for the server-side portion and “3 - doable” for the client side portion. The combined value gives the value of the user story, in this case 5.
From these point values, the project manager can then go back to the client, and ask, “With 20 points, what features do you wish to have completed in the first Sprint?”
The client can “purchase” user stories using those 20 points. A Sprint, which I elaborate on below, is an uninteruppted time period in which your team works on completing their assigned user stories. You make a guess in Sprint #1 on how many points your team can complete per Sprint. As time goes on, you start to see a pattern emerge of how many points per Sprint is the team completing. The client and managers can use this metric for planning and resource purposes. You can forecast, somewhat, how many Sprints you have to do to complete 60 points worth of features, and when you’ll have them completed. The chosen user stories that the client purchased are what your team is working on in Sprint #1.
Assigning points is a very important step for the engineering team. You typically don’t change point values mid-project, only assinging points to modified user stories or new user stories.
As I mentioned, Sprints are a period of time that your team works on their assigned user stories. A sprint is however long your team decides it should be to get a working build with the user stories assigned. This could be one week, two, or a month. My current team is using 2 week cycles. It’s important to note the “uniteruppted” part. The client and project manager won’t interject user stories mid-sprint, nor modify your work load. Basically, they don’t bother you. This allows your team to focus on the task at hand, prevent fire drills, and stay productive.
In the examples provided, your team has 2 weeks to complete 20 points worth of user stories. Now, a sprint isn’t just, say 10 days of straight working. You usually have 3 special days. The first day is the kick off. You determine the user stories you and your team members will tackle. A couple days before the last is your “merge day”. It’s when everyone merges the code together from their different branches (or you just make sure trunk works if not using branches, more on that later) to get an idea of where the build is at, and where your team is at with their user stories. This is a final checkpoint before…. the UAT. UAT stands for user acceptance testing and in this case, it’s going over the final working build. Your team collectively goes over each user story, and identify if the build satisfies the user story or not.
As I said, our sprints are 2 weeks in length. The first Monday is spent confirming what user stories are in the current sprint and who’s working on them. We try to work in our own personal branches so the 2nd Wednesday is “Merge Day”, where I take any branches I and my teammate have, merge into trunk, and work out the kinks. We make a mad dash the rest of Wednesday and all of Thursday to finish our remaining user stories, or fix issues with existing ones, or just the build in general. Again, over the course of a sprint, you can start to get a group average of how many points your team is capable of doing in a sprint.
We’ve been using a shared Google Spreadsheet with each Sprint’s user stories in its own tab. We collaboratively use this all the time for reference.
We have a call everyday at 1 pm EST / 10 am PST. Our server-side time + client is out west in Cali, and our Flex team + UX + designer + project manager + rest of the team for other areas is east in Atlanta. We each go around talking about what we completed yesterday, this morning, and what we plan on going the rest of the day. We also cite any roadblocks or issues, and if that involves another team member, we just schedule a separate call, or time to talk over IM, with that specific person. This results in our calls usually lasting 5 minutes each day. I’ve heard of other projects at Enablus having their daily standup lasting longer than 5 minutes, but our team is pretty focused; we know what we need to do.
The majority of my road blocks involve a web service not working like expected suddenly, or questions for the designer on how a part of the design comp is supposed to work. Most of these I just solve on my own by calling the team member outside of the meeting time.
Pros & Cons
There you have it, what Agile is based on what I’ve been told and experienced so far. So how’s it working out so far? Here are the pro’s and con’s for this first entry in the series. I’ll elaborate in future articles, but here’s the quick rundown.
- I’ve never been this stressed in the first two weeks of a project before.
- coding with a “git-r-done” mentality; fast and furious. Not everything has to be uber-architected.
- each developer has their own branches; this makes checkins & merges a lot easier and less stressful
- It’s nice to have something real after just 2 weeks. Gives a great feeling of validation very quickly
- a lot of re-factoring
- creating new user stories later can break older ones
- merge day is stressful
- “need to see it working”; the desire to see working functionality before important functionality decisions can be made isn’t solved
In conclusion, I really like Agile methodology that we’re using so far as it allows me to code some things extremely fast and not have to worry about uber-architecting it. I just have to get it working which I’m good at doing quickly. For the things that matter, I can spend an allocated time which I feel good about doing because I can more easily see over time which areas deserve architecture and which ones do not. The merge day is extremely stressful, but overall I like having an allocated day for it and using separate branches which prevent daily SVN drama.
I’ve never been this stressed in the first two weeks of a project before. The typical scenario is you get excited, and can’t wait to dive in. It’s a happy feeling filled with anticipation. Then, as the deadline approaches, you steadily get stressed out and things become less fun, albeit extremely frustrating. This isn’t always the case, but I’ve continually expected fun in the beginning, stress at the end.
It’s been really nice to be stressed at the beginning; this has implied to me the project won’t be so traditionally bipolar. The same thing happened to my mood once I capped myself at 2 cups of coffee a day. Think of it, you have 2 weeks to create working software; ready, GO! That’s insane, and really cool at the same time. What could you do in 2 weeks? It’s really nice to know that even after just 2 weeks of work on a 4 month (ish) long project, even the first 2 weeks will produce something valid and working.
Stay tuned for #2 in the Agile Chronicles series where I elaborate on the re-factoring challenges.
Cloud computing is being adopted in one form or another by 94% of enterprises today. Tens of billions of new devices are being connected to The Internet of Things. And Big Data is driving this bus. An exponential increase is expected in the amount of information being processed, managed, analyzed, and acted upon by enterprise IT. This amazing is not part of some distant future - it is happening today. One report shows a 650% increase in enterprise data by 2020. Other estimates are even higher....
Jun. 26, 2016 05:00 PM EDT Reads: 1,284
SYS-CON Events announced today that Bsquare has been named “Silver Sponsor” of SYS-CON's @ThingsExpo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. For more than two decades, Bsquare has helped its customers extract business value from a broad array of physical assets by making them intelligent, connecting them, and using the data they generate to optimize business processes.
Jun. 26, 2016 05:00 PM EDT Reads: 1,198
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, wh...
Jun. 26, 2016 04:30 PM EDT Reads: 927
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 19th Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - comp...
Jun. 26, 2016 04:00 PM EDT Reads: 1,236
19th Cloud Expo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Meanwhile, 94% of enterpri...
Jun. 26, 2016 04:00 PM EDT Reads: 1,284
Connected devices and the industrial internet are growing exponentially every year with Cisco expecting 50 billion devices to be in operation by 2020. In this period of growth, location-based insights are becoming invaluable to many businesses as they adopt new connected technologies. Knowing when and where these devices connect from is critical for a number of scenarios in supply chain management, disaster management, emergency response, M2M, location marketing and more. In his session at @Th...
Jun. 26, 2016 03:00 PM EDT Reads: 942
It is one thing to build single industrial IoT applications, but what will it take to build the Smart Cities and truly society changing applications of the future? The technology won’t be the problem, it will be the number of parties that need to work together and be aligned in their motivation to succeed. In his Day 2 Keynote at @ThingsExpo, Henrik Kenani Dahlgren, Portfolio Marketing Manager at Ericsson, discussed how to plan to cooperate, partner, and form lasting all-star teams to change t...
Jun. 26, 2016 02:00 PM EDT Reads: 1,122
The cloud market growth today is largely in public clouds. While there is a lot of spend in IT departments in virtualization, these aren’t yet translating into a true “cloud” experience within the enterprise. What is stopping the growth of the “private cloud” market? In his general session at 18th Cloud Expo, Nara Rajagopalan, CEO of Accelerite, explored the challenges in deploying, managing, and getting adoption for a private cloud within an enterprise. What are the key differences between wh...
Jun. 26, 2016 01:00 PM EDT Reads: 754
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
Jun. 26, 2016 12:00 PM EDT Reads: 1,279
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo Silicon Valley Call for Papers is now open.
Jun. 26, 2016 12:00 PM EDT Reads: 1,100
There is little doubt that Big Data solutions will have an increasing role in the Enterprise IT mainstream over time. Big Data at Cloud Expo - to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA - has announced its Call for Papers is open. Cloud computing is being adopted in one form or another by 94% of enterprises today. Tens of billions of new devices are being connected to The Internet of Things. And Big Data is driving this bus. An exponential increase is...
Jun. 26, 2016 12:00 PM EDT Reads: 1,333
In his general session at 18th Cloud Expo, Lee Atchison, Principal Cloud Architect and Advocate at New Relic, discussed cloud as a ‘better data center’ and how it adds new capacity (faster) and improves application availability (redundancy). The cloud is a ‘Dynamic Tool for Dynamic Apps’ and resource allocation is an integral part of your application architecture, so use only the resources you need and allocate /de-allocate resources on the fly.
Jun. 26, 2016 12:00 PM EDT Reads: 1,068
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life sett...
Jun. 26, 2016 11:00 AM EDT Reads: 1,075
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning techniques such as Cognitive Fingerprinting, wind project operators can utilize these tools to learn from collected data, detect regular patterns, and optimize their own operations. In his session at 18th Cloud Expo, Stuart Gillen, Director of Business Development at SparkCognition, discussed how research has demonstrated the value of Machine Learning in delivering next generation analytics to imp...
Jun. 26, 2016 09:45 AM EDT Reads: 593
There are several IoTs: the Industrial Internet, Consumer Wearables, Wearables and Healthcare, Supply Chains, and the movement toward Smart Grids, Cities, Regions, and Nations. There are competing communications standards every step of the way, a bewildering array of sensors and devices, and an entire world of competing data analytics platforms. To some this appears to be chaos. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, Bradley Holt, Developer Advocate a...
Jun. 26, 2016 08:45 AM EDT Reads: 728
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
Jun. 25, 2016 03:00 PM EDT Reads: 1,590
SYS-CON Events announced today that ReadyTalk, a leading provider of online conferencing and webinar services, has been named Vendor Presentation Sponsor at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. ReadyTalk delivers audio and web conferencing services that inspire collaboration and enable the Future of Work for today’s increasingly digital and mobile workforce. By combining intuitive, innovative tec...
Jun. 24, 2016 01:00 PM EDT Reads: 1,349
Amazon has gradually rolled out parts of its IoT offerings, but these are just the tip of the iceberg. In addition to optimizing their backend AWS offerings, Amazon is laying the ground work to be a major force in IoT - especially in the connected home and office. In his session at @ThingsExpo, Chris Kocher, founder and managing director of Grey Heron, explained how Amazon is extending its reach to become a major force in IoT by building on its dominant cloud IoT platform, its Dash Button strat...
Jun. 24, 2016 12:00 PM EDT Reads: 1,627
industrial company for a multi-year contract initially valued at over $4.0 million. In addition to DataV software, Bsquare will also provide comprehensive systems integration, support and maintenance services. DataV leverages advanced data analytics, predictive reasoning, data-driven diagnostics, and automated orchestration of remediation actions in order to improve asset uptime while reducing service and warranty costs.
Jun. 22, 2016 11:00 AM EDT Reads: 1,368
Vidyo, Inc., has joined the Alliance for Open Media. The Alliance for Open Media is a non-profit organization working to define and develop media technologies that address the need for an open standard for video compression and delivery over the web. As a member of the Alliance, Vidyo will collaborate with industry leaders in pursuit of an open and royalty-free AOMedia Video codec, AV1. Vidyo’s contributions to the organization will bring to bear its long history of expertise in codec technolo...
Jun. 19, 2016 12:45 PM EDT Reads: 1,266