Protocols Of Matrix: Interesting Article found by Matt Gullett.

Source :http://www.codeproject.com/gen/work/standaloneinnovative.asp

The Standalone Programmer: Innovative Software Development
By Matt Gullett.

Introduction

Have you ever noticed that sometimes a single programmer (or very small team) can accomplish more than a large team? Sometimes it seems that large teams are totally unable to meet deadlines or when they do, they deliver mediocre code. Why is this? I believe that the reason for this is that software development is very different from most other jobs within most companies and most companies do not recognize this. Companies have become very good at mass producing everything from textiles to jet aircraft. The obvious thing to do is to create an assembly line and place people (workers) at various strategically chosen places on that line. Each worker is responsible for a specific task and quickly masters whatever that task is. This mentality has proven problematic in fields where innovation and talent are key. Consider innovative work like designing and building the space shuttle, missions to mars, building more efficient fuel sources, etc. In any of these fields, you will find cost overruns, missed schedules, and all the problems that are found in software development. Why? Because these fields require innovation, not automation. It is not about mass producing something, it is about inventing something totally new.

Let's face it, software development is about more than following rules, knowing the syntax of a language, following a development methodology or anything else. Software development can be amazingly simple or incredibly difficult. The only thing that can set a standalone programmer apart from the rest of the development community is his or her ingenuity, raw talent and experience (and in the absence of either, really bad management.) Leveraging that ingenuity can sometimes seem impossible because of the responsibilities we have on our job, and the constraints placed on us by ourselves and others. These responsibilities and constraints stifle innovation and can prevent break through ideas from seeing the light of day.

At some point, every one of us will be faced with a project or problem which when analyzed systematically appears to be unachievable, prohibitively time consuming, expensive, or complicated. At this point we often turn to off-the-shelf solutions to try and solve the problem or decide that we need to hire more programmers. While this may be the best possible approach, it is often a mistake to jump to this conclusion without first "getting away from the problem" and "thinking outside the box". The time has come to put that ingenuity, talent and experience to work.

The problem with education

There is a fatal flaw in the way people are taught to "program" in college courses and the software press. The same approach to software development is applied to every problem. Sure, there are different development methodologies, languages, tools, etc, but tools, languages and methodologies are not enough. Education simply cannot create talent where there is none. The idea that you can teach people to think outside the box and innovate on their own, while an admirable goal, is just not realistic. What education can do is expose talented people to new ways of doing things and provide a level of experience (though usually quite limited) that can provide a spring board for future success.

For some reason, many businesses fail to understand this and continue to place too much importance on college degrees, certifications, etc. If you read any business-oriented IT magazine or website, you will be inundated by reports of failed projects, statistics on cost overruns, missed schedules, etc. Companies are continuing to apply the same criteria to IT that they have for years, yet they expect different results. To me, this is ludicrous. The "if at first you don't succeed, try, try again" motto is being applied blindly without considering past performance and searching for better ways.

What needs to be done? If I knew the answer I would probably be sitting in Hawaii right now instead of Winston Salem, NC.

There is a flip-side to this problem, too. Many talented software developers devalue education to their own detriment. I do not have a college degree and this is something I regret all too often. If I could go back to when I was 18 and single, I would take the time to acquire a degree. There was a time when I said that I could learn anything I needed to without going through formal education. This may be true, but it would probably be much (much) easier to learn some things in a formal environment. My advice to those of you who are currently enrolled in a formal CS program and considering quitting, is to stick it out and learn as much as possible as you can.

Breaking the rules

I doubt that Shawn Fanning spent much time performing unit tests, usability studies or performing code reviews when he developed the first version of Napster. Maybe he did, but I seriously doubt it. To me, his story is one of innovative software development. He had a good idea and a commitment to seeing his idea brought to completion. I also doubt that Bill Gates and Paul Allen wrote a detailed design document before writing BASIC. They had an idea and free time and created something totally new.

These stories are not typical, but they are well known examples of something that happens every day on a smaller level. With every new project or problem, there is opportunity for innovation. Each situation requires that software developers use their talent and experience to evaluate the problem and design and develop an appropriate solution. A systematic process of design and review is always needed and should never be omitted. However, sometimes there is a nagging little voice that says "there's a better way to do this". When you hear this voice, stop and listen to it. Take a different perspective, try different things on a small-scale, scour the web, and don't give up until that little voice is satisfied. You may discover an approach you had not considered before.

Now you have a problem. How can you properly design something that is just an idea? Should you jump in and start slinging code, or should you be systematic and mentally test every eventuality, review every issue and write up your results? I am going to go against conventional wisdom, and say jump in and start coding. Start up that IDE and put that talent and skill to work and don't stop coding until your idea is vindicated or decimated. Don't look up, don't go to lunch, don't do anything that distracts you from the idea.

Some CS major or professor out there is screaming at the top of their lungs right now, saying "DON'T LISTEN TO THIS GUY!". Don't ignore all the screaming, but think of it as a control valve. There are always 2 extremes to anything. You can go overboard by doing too much or too little. All those rules and guidelines are there for a reason. They can and do work for most things. Don't read this article and use it as an excuse for ignoring the rules and guidelines all the time. I am not writing an "opposing view", but an additional view which can be combined with the rules to achieve greater success.

Rules to break and rules to keep

OK, so you have an idea, you have the IDE up and running and you are just about ready to start banging on the keyboard. Jump to it. While you are slinging all that code, here are a few tips to remember which can make you more productive and help keep your idea (if vindicated) viable.

Don't forget code quality guidelines such as initializing variables, code style, etc. These basic guidelines don't take any time if they are already habit anyway. Why, you may ask? Simple, you are going to have to test the code you have written and if it works, don't kid yourself, some of that code will find its way into production.
Be realistic. Rome wasn't built in a day.
Be unrealistic. Contradictive, isn't it? Think "outside the box", nothing is impossible. Rome wasn't built in a day, but maybe if your innovation existed, it would have been.
Don't forget comments. Your brain is converting thoughts into code. All to often when working without a plan, the thoughts only flow one way. You will write a function, class or something and the next time you look at it, not remember anything about it. I think of comments as code anyway, so commenting is just part of the coding process.
Keep your issue tracking software up-and-running while you code. (You do have an issue database don't you?) Use that issue database to keep track of the ideas that you have. If you don't have an issue database, then create a comments.h file and keep your ideas in there.
Don't worry too much about the structure of your code. You will probably write a function which should be broken up into several or a class which will need renamed, or rebased. Don't worry about these details much. What you want to achieve here is not perfect code, but to vindicate your idea.
Don't watch the clock.

Knowing when to stop

The goal of innovative development is not to build the world, but to determine whether your idea has merit and is feasible. Be prepared to find out that your idea won't work. Be prepared to toss your days work in the trash and never look at it again. So, at what point do you stop trying and go back to the systematic approach?

When you hit a roadblock, which seems insurmountable, STOP. Stop coding, stop working, stop looking at the code, stop thinking about the code, shut down the IDE and get away from the problem. Your idea is not dead, but you have to solve this problem before continuing. Why stop coding? Because your current mindset is to solve the problem through code and that is not what you want to do. You want to solve the problem with your brain. Your first thought will be to skip this problem and come back to it. This is very dangerous, because you will forget about the problem, or worse, solve the other 99 problems and leave this one insurmountable one behind. All the work on the other 99 issues, while not a total loss, will probably be a waste of time because you won't be able to use that work anyway.

When you reach a stopping point, evaluate where you are, what you have achieved and what remains to be done. Sometimes we programmers pretend that things are easier than they actually are. So, be fair and use your best judgment to gauge your progress. There is no magic formula, but experience will make this easier and hopefully more realistic. I have found that taking a lunch or dinner break alone and self-evaluating the work is very effective. I often come up with new ideas or realize something I missed.

How much time you spend in this innovative development cycle is also guided by experience and self-evaluation. You may spend minutes, hours, days or even weeks working like this. My advice to young programmers is to keep these sessions relatively short and never more than a day. There is a great deal that can be learned from many one or two hour sessions that is often not possible from longer sessions. Use these shorter sessions as learning experiences and opportunities to experiment while limiting your risk. The bottom line is recognize your own limits. Push yourself, but be conscious of your own abilities, experience and talent.

On the flip-side, you may discover that your idea works out perfectly. When you reach this conclusion, STOP. Stop coding, stop looking at the code, stop compiling the code, close the IDE. Why? Because you have achieved the goal of innovative development. You have solved your problem, vindicated your idea and are ready to leverage that idea for your needs. I strongly recommend that the first thing you do at this point is to write down what you did and how you did it in Word or notepad or something. Be as detailed as possible and try to cover all the important topics.

Two heads are better than one

I have found that bouncing my ideas off of a coworker is an excellent way to flush out the good ideas and filter out the bad ones. I am blessed with a boss who is an ex-programmer with excellent technical skills and knowledge. It is not uncommon for me to walk into his office and throw out some crazy idea I have had. Sometimes we will sit and discuss the idea for a long time before reaching a conclusion that it is infeasible or problematic. Sometimes he will say (in kinder words), that my idea is stupid. These brain-storming sessions often result in more ideas coming out because both of us are working on the problem together. Often, the original idea I had is proven feasible, but an even better solution is found, and sometimes, we determine that the original idea was sound and feasible.

One of the keys to this part of innovative software development is that the lines of communication must be open and egos have to be set aside. You may find that whoever you are working with knows something you did not or has encountered the problem before, or simply has some insight that you overlooked. You must not be protective of your ideas to a fault and you must be willing to accept that his/her ideas are better than your own.

I would advise every young programmer to seek out a mentor who can help you by providing support, experience and his/her own unique perspective. A mentor need not be a programmer, though it may help. Look for someone you can trust and feel comfortable talking to. Make sure that you are comfortable enough to have your ideas discredited. Your mentor must also be someone who can set his/her ego aside and see the big picture.

Having someone to bounce ideas off of and mentor you is a wonderful asset that can be priceless. You will eventually encounter a problem that is beyond your experience or skills, but having someone to talk to can help you to expand your experience and skills with fewer headaches and set backs.

Some of you may be thinking that this goes against the grain of the standalone programmer. I disagree. I am not saying that this other person will do your job, I am just stating that you will be a better developer if you can expose yourself to other peoples ideas and concepts. You will also find that you will make fewer mistakes in the long run.

There is nothing new under the sun

So, you have vindicated your idea and know exactly how to make use of it. When I reach this point, I think it is a good idea to bring up my trusty web browser and head straight to google. You may find that someone else has already encountered the same problem and come up with the same solution. Some of you may say that this is something that you should do before you jumped into the code. In this case, I believe they would be wrong. By allowing yourself to experiment and completely flush out all the issues, you are better able to review existing options and know what things to look for. You are much more likely to see problems in their design or implementation and be ready to ask probing questions (when possible) to their sales people/engineers/coders. You may find that an existing solution completely meets your needs and does exactly what you want it to. It may be a better business decision to use this existing solution than to take your idea and turn it into a production solution.

You can also use this as an opportunity to flush out any problems or issues you may have overlooked. When reviewing existing solutions, compare the details of the implementation and design to your own. Is there something they deal with that your solution does not? Be thorough in your review. The whole reason for taking this innovative approach was to solve a seemingly insurmountable problem. This is no small task and should never be underestimated. It is easy to overlook something and that is not what you want.

Even if you find an existing solution, you may still decide that you need to implement your own. Maybe the existing solution is too expensive, written in a language you are not familiar with, does not provide source code, etc. Maybe, you just think that you can do a better job. Innovate. Do the best thing you can based on your knowledge of the problem and solution.

What to do now

You have had an epiphany, tested and proven your ideas and determined that you need to develop a solution based on those ideas. Great! Remember those rules we have been ignoring, well it's time to pick that rule book out of the trash and turn to chapter 1. Requirements Analysis. That's right. You're not done yet and you can't just ignore those rules. However, having done all the fun stuff you just finished, you are better prepared to apply those rules. You will most likely be able to fly right through the process and jump right back into the coding because you are intimately familiar with the requirements and the solution. Don't skimp because there is always more to a project than just an idea and some code. You need that process to insure that you don't miss something, don't do too much and ultimately deliver high quality solutions.

All that code you wrote during your code slinging sessions may not need to be rewritten. If you followed good code quality guidelines, you may be able to refactor (OMG! I can't believe I just used that word.) that code into the production system. Don't be fooled into thinking that it is ready for prime time, because most of the time, it is not. The purpose for that code slinging was to let your talent flow into code, not into software. Know the difference and know how to take one and make the other.

Reality bytes

The idea of bypassing all that boring process stuff (design, requirements gathering, etc) to exercise your coding skills sounds wonderful, but most of us don't work in pure research environments. We work for companies which have real business needs and/or demands. You may be a freelancer in which case you still have to pay the bills by providing meaningful solutions to your clients. If your company has a well defined software process, you may have less flexibility in terms of choosing how you solve a problem. As a freelancer time is money and sometimes choosing a well trodden path is a much less risky choice than trying to do something new and innovative, and sometimes there just are not enough hours in the day.

Breaking away from the constraints of reality can be difficult. Even if you find a way to break the rules, you may not have an idea "ready to go" or you may think that your idea is too simple or has been done before. If you don't have an idea, think about the problems you or your users face on a regular basis. Take a look through the issue database (you thought I forgot, didn't you?). What about the project you are currently working on? Surely there is something that you can think of that needs improvement. Dwell on the subject for a while. Bounce a few ideas off a coworker, or ask a coworker for ideas.

It is likely that the idea you come up with will not seem like a tremendous achievement. So what! It is your idea and who cares if someone has done it before. You know you can do a better job that everyone else, or at least a better job than those around you. Skip a lunch, do whatever it takes, but find some time to try your idea out. If you feel the need to research the idea a little, do that too, but don't let yourself get distracted. Maybe you just can't find time to try it out. Write it down, then. Write as much as you can because if you don't, you will forget.

Once you have proven your idea, you may need to find a way to get it used in your companies product. You will need to prove your idea to your manager, team leader, boss, client or whatever. Keep this in mind while you are slinging all that code. Don't forget that for your idea to see the light of day, no matter how good it is, you will have to sell it to someone. What makes that person tick? How can you sell your idea? Will it make the product development go quicker? Will it reduce the cost? Will it increase sales? (If so, how?). Will it help them get a promotion or raise? Don't forget the impact it will have on you either. How can your idea advance your career, help you get a raise or whatever it is you want?

Hard lessons learned

Before I drop the subject of innovative software development, I want to instill in you the fact that sometimes the results are not as good as you thought they would be. To illustrate this, I am going to discuss one of my own failed projects. (NOTE: Although I was the sole programmer for this project, my boss and I worked together to design this project.) Unfortunately, reality bit me because I failed to solve 100% of the problems with my little innovation. A couple years ago, I had a client (while working for a previous employer) that needed a very in-depth database application. This particular client is a non-profit organization which provides various services to early childhood educators and day care workers. The goal of these services was to improve the quality of childcare by improving the quality of the workforce involved in childcare. The purpose of the application was to track all of the activities that the organization had with regards to the early childcare workforce and track the progress of individual work force members, as well as the workforce as a whole, over time.

I can't begin to explain every detail of the system, but here are a few details. Within the organization, there were several different departments which each provided a different set of services. Each of these departments collected various pieces of information from the workforce (often the exact same data collected by other departments), and performed different tasks. Some departments held workshops, others provided financial assistance, some provided in-class training or observation, some did testing, The application required that all interaction with workforce members be entered into the database. It also required that all data collection efforts be handled through the database so that there was a complete record of all activities. Every user needed a different user interface to accomplish his/her task because they were largely different. On the back end, the status of workforce members had to be tracked over time and reported on in many different ways.

At first, the task seemed insurmountable. How to provide different UIs to many different users, with different features and goals, while utilizing the same back end database and capturing a common set of data points? This problem and many more were overcome through a process of innovative development and practical development process. The end product provided separate UIs, shared the database, unified the data collection efforts, reduced confusion, and provided meaningful and in-depth reporting on the progress of change and improvement within the workforce. There were quite a few bumps and problems the first year gaining user acceptance and working out the kinks, but it seemed that everything was going well.

That didn't last long. Way back in the early stages of development, I had to come up with a way to unify the database on the back end while providing completely different UI's and features on the front end. This was just one of many problems that were overcome, but this is the one that ultimately doomed the project. The solution I came up with was extremely flexible and made it possible to store just about any thing that might be needed and merge that information with the appropriate workforce data. I knew of only one problem with the solution, and it became a critical flaw. The solution made it easy to get data into the database, and facilitated back-end historical reporting, but front-end day-to-day reporting was just not easy and just about required an advanced degree in SQL and a magic wand. I was aware of this problem and I even developed a half-ass solution to it, knowing full well that it was not a 100% solution.

The result was a system which met all of the customer requirements for historical tracking, multiple user interfaces, unified database, and much more, but lacked one critical component. Front-end user reporting was a royal pain and was often just not possible without going through many convoluted steps which only I was really able to do. As of today, this project is not completely a failure, but is headed down that ravine rather quickly.

The reason for sharing this little story is to stress the importance of solving ALL, 100% of the problems with your innovative solutions, because the client needs a 100% solution, not a 99% one.

Conclusions

Sometimes all the "proper" development methodologies and rules interfere with real advancements and innovations. As a standalone developer, it is important to recognize this fact and know when to break with tradition and exercise your talent and skills. It is just as important to realize that the rules and methodologies can have a tremendous impact on software quality. Leveraging both techniques will make you a better developer and may just make you the next Bill Gates.