How to merge several Excel worksheets using RapidMiner

This is a tutorial in using RapidMiner 5.3 ETL features to merge several Excel worksheets from the same file into one single output which can then be saved into a separate Excel file, a CSV file or database table.

RapidMiner (http://rapid-i.com) is the world-leading open-source system for data mining. It is available as a stand-alone application for data analysis and as a data mining engine for the integration into own products. It has an awesome array of ETL tools that lets you work with any data quickly and efficiently. The Community Edition is free of charge which makes it perfect for use in this task.

Introduction

RapidMiner uses a vast collection of customizable data operators to design a process flow for your data. I’ve used RapidMiner before to merge several CSV files into one using its Loop Files parameter, but in this case the data was all in one excel file separated into several tabs, so I needed to go a little deeper. This tutorial assumes some knowledge of RapidMiner although it’s really easy to learn.

There is another operator called Loop Parameters, which you can find by typing it in the search box. It allows you to loop through the parameters of any operator.

Untitled picture

The little blue icon on the bottom right indicates that this is a container, inside of which we need to  place the actual action operator that does the work. So double click on the Loop Parameters operator to go inside it.

Now we need to add our action operator, in our case it’s Read Excel. Search for it and drop it into the palette. If you’ve set RapidMiner to auto-wire inputs, it will automatically link the output from the operator to the output wall.

2013-09-29_2120

There are two things we need to customize:

First let’s customize the Read Excel operator’s properties. We will need to specify the file that we’re reading from. I’ve created a dummy file (which you can get here) for the purposes of this tutorial; the data in it doesn’t mean anything. The file contains 3 sheets with sales data.

Sheet1

Division Sales
1 $3,400
2 $3,500
3 $3,600
4 $3,700
Sheet2

Division Sales
4 $4,100
5 $4,200
6 $4,300
7 $4,400
Sheet3

Division Sales
9 $4,600
10 $4,700
11 $4,800
12 $4,900

Our goal is to merge the data from all three sheets into one single sheet.

Note: In order for this to work properly, ALL the sheets need to have the same exact headings. If they don’t have the same exact headings, you have to take care of that first, otherwise the merge will not work.

Click on the Read Excel operator and then on the Import Configuration Wizard button on the properties panel on the right hand side.

2013-09-29_2138

The wizard will guide through configuring the input file. In Step 1 you choose the directory where you saved the file, click on the file and then click Next.

2013-09-29_2140

In Step 2, RapidMiner will load the Excel file so you can see all the Sheets and go through them to ensure they all have the same headings. Ensure you can see  Sheet1, Sheet2 and Sheet3. Click Next.

2013-09-29_2142

In Step 3, you can annotate a column with special properties, in our case the column headings. RapidMiner does this for you, so no action needed. Click Next to proceed to the final step.

2013-09-29_2144

In Step 4, you can choose the data types for your columns. RapidMiner will try to guess the values for you and it usually does a good job, but if you want to change them, just click on the down arrows for each column and pick the data type you like. When done click Finish.

2013-09-29_2146

Next, we need to specify the settings for the Loop Parameters operator. To do this, click out of the Read Excel operator (by clicking anywhere else on the palette) and the properties box should change as shown: At this point we are till inside the Loop Parameters operator

2013-09-29_2150

Click on the Edit Parameter Settings to continue. We get the following dialog box.

2013-09-29_2153

There are several things going on here. Since this is a generic loop container, we need to specify the operator(s), the parameters to loop through and the range of values. In our case, we only have one operator (Read Excel) Click on it and you’re presented with the parameters that we can choose to loop. We want the sheet_number parameter. Click on it and then click on the right green arrow (==>) to add it to the Selected Parameters box. Your screen should look like this:

2013-09-29_2157

Next we need to specify the range of values we want the parameter to take. We only have 3 sheets so choose Min = 1; Max = 3; Steps = 3; Scale = Linear. When done, click Ok. There are more options here for advanced looping, but this is outside the scope of this tutorial.

We are now ready to test our work! The nice thing about RapidMiner is that you can build your process in steps and check your work at any step by running it and seeing your results. To do this we need to first exit the loop container by clicking on the blue Up arrow to get to the main screen.

Untitled picture 1

Once in the main screen, connect the Loop Parameter‘s res (ResultSet) port to the res port on the output wall. Then click on the blue triangle on top to run the process.

2013-09-29_2210_002

If everything was set up correctly, RapidMiner will execute the process and present you with the results as shown here:

2013-09-29_2213

We can see in the Results Overview an IOObjectCollection tab with all three sheets as separate ExampleSets (RapidMiner calls its outputs ExampleSet. What we now have is a collection of the data from the three sheets but it’s still separate, so we need one more operator to do the actual merge.

Switch back to the Design perspective and add the Append operator. If you’ve set RapidMiner to auto-wire the operators and if you hold the Append operator right over the purple output line, it will drop it in place and be automatically wire it. If not, you have to manually wire it.

2013-09-29_2217

Run the process again and now we get the results we want! The ExampleSet from the (Append) operator shows the data from the three sheets merged into one. Nice!

2013-09-29_2221

At this point we can add an export operator to save it somewhere. RapidMiner supports a variety of export options: there’s separate Write operators for Excel, CSV, database tables, etc.

2013-09-29_2226

We are done. If you have any questions, please put it in the comments.

Patterns of Strategic Thinking – The Secrets of Exploiting Leverage

Introduction

I’ve recently been obsessed with strategy and have been reading a variety of strategy books in a variety of fields such as psychology, economics, business, military, etc.  I noticed that while few people really understand strategy (even though they think they do) there were certain patterns of strategic thinking that show up over and over again in these works. These patterns can easily be abstracted out of their respective fields and used in their everyday life.

Game theory is one of the fields that studies some of these patterns from the perspective of games where you and an opponent take turns making moves. According to Wikipedia game theory is: “the study of mathematical models of conflict and cooperation between intelligent rational decision-makers.”

Your objective in these games is to be able to predict what moves your opponent will make for every one of your moves and choose the most optimal one for you to win the game (however winning is defined)

Personally I’m not a big fan of game theory for two core reasons. First it limits your thinking to a frame of move-countermove when in real life things are a lot more dynamic and complex. In fact, in many cases you can build quite an advantage without worrying about what your opponents or competitors are doing. Second it utilizes a model of the “intelligent rational decision makers” which according to many studies in behavioral economics is incorrect.

I first started to notice these patterns while I was studying psychology and influence, however I saw them as specific techniques within those fields. It wasn’t until I started reading books on strategy that I was able to generalize them and abstract them out of those fields and I started to see them everywhere. The techniques were different of course but the patterns were eerily similar

For example you can leverage certain skills you have to get a higher position in one company than you’d be able to get in another company. In another setting, say in business, you might be able to leverage big cash reserves to invest in R&D to move your company beyond the current trends in technology and be able to actually shape the future rather than be shaped by the future.

In both cases you’re leveraging/exploiting certain assets, that are only available to you, to put yourself in a better position to succeed. What else can your leverage that puts you in a position of advantage against your competition? More on that in a bit.

This is the first in a series of posts on the patterns of strategic thinking. My goal with this is to create a collection of these patterns that can be used as a toolkit for making smarter decisions both in business and in personal life and who doesn’t want to make smarter decisions?

In this post we will explore the vast and unending ways you can exploit and make use of leverage.

Using Leverage

“Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.” – Archimedes

Leverage is simply a way to multiply the force used to a greater effect or to achieve a specific objective. It’s a type of advantage that is context free and not rooted in any particular field.

You can get or use leverage in one of two ways:

  1. by exploiting certain patterns that exist in a system either innate to it or by design
  2. by creating and accumulating assets which can be used at a later time

Let’s explore these two in depth.

There are three categories of patterns you can notice and exploit in systems:

  1. Predictive Patterns
  2. Pivotal Points
  3. Focused Effort

Part 1 – Exploiting Patterns

1. Predictive Patterns are patterns that allow you to predict what might happen in the future through the use of trends in the industry, momentum, routines, habits, biases, social dynamics in a given context, psychological models of behavior, inertias, etc.

In game theory a lot of your leverage and advantage comes from your ability to predict other people’s responses to your moves and figure out the countermoves to that anticipated behavior so you can win the game in the end. The farther you’re able to predict, the more likely you are to win the game.

As I mentioned above, one of the issues I have with it is that it assumes that you’re dealing with intelligent and fully rational human beings. However, as Daniel Kahneman and Amos Tversky (and later Dan Ariely in Predictably Irrational, Richard Thaler and Cass Sunnstein in Nudge and many others) have repeatedly pointed out, humans deviate from the so-called “rational behavior” but do so in relatively predictable ways.

So you can either use the “rational model” of behavior, or you can use the biases model that behavioral economics has put forth. In fact a lot of the techniques that are discussed in books about persuasion and influence focus specifically in telling you the various patterns of behavior that humans tend to follow which can be used for leverage.

Here are a few examples.

-In investing/business, if you notice a trend towards cloud computing and anticipate growth, you can invest in a company that owns and operates a lot of data centers.

-In your career, if you notice that everyone is talking about data science and big data and anticipate a big demand for people who know data analysis, you can start to learn statistics, if you already know programming or programming if you already know math/statistics.

-In your personal or social life, if you notice that when you call yourself a statistician people stop wanting to talk to you, next time you can call yourself a data scientist and get them more interested.

2. Pivotal Points are things like imbalances, inefficiencies, weaknesses, and so on that magnify the effects of effort. They are either innate to the system or are put there “by design” and once noticed can be used over and over again until the system is either corrected or corrects itself.

I put “by design” in quotes to highlight the fact that in many cases the weakness that are exploited in certain systems, (for example networks or websites that are hacked) are usually not there on purpose but through design they were introduced in the system as bugs probably through oversight or just faulty construction.

Imbalances exists when a small shift in the system results in relatively large effects. This can be for example pent-up demand for an item that nobody thought would be there. In war/conflict an imbalance would be a perceptual difference between say the number of soldiers that the enemy says they have and their actual numbers.

Inefficiencies are usually extra steps that one has to take to accomplish something in the system and are the most easily noticed form of leverage. Google for example noticed that the big search engines operating in the late 90’s all had the same inefficiency. They would provide irrelevant search results and as an advertiser you might be able to buy yourself into the top of the results list and stay there as long as you were paying.

Google exploited this inefficiency by introducing their PageRank algorithm and later their bid-style PPC ad buying platform. This not only had the effect of millions of users switching over to Google as their default search ending but also leveling the field for everyone so the little guy could compete with the big advertisers on an even footing.

Notice that this was both an imbalance (search engine users wanted to see relevant results so there was pent-up demand) and an inefficiency (users had to scroll through many irrelevant search results to get to what they were looking for sometimes never finding it)

Weaknesses are areas of lack of competency by an opponent/competitor that can be exploited and used by someone who has that competency. Of course the actual competency needs to be in demand in order for it to be worth exploiting. Again this is another common way that leverage is used in business or even personal life.

A good example of this is choosing a career. We’re all good at some things but not others, so for example if you find yourself in an organization that has really poor processes that are hindering growth and you are good at project management, this is a weakness you might be able to exploit long enough to either become an expert in project management or get promoted within the organization.

The examples above should be plenty to stir your imagination so let’s move on to the third pattern.

3. Focused Effort is a pattern where you limit your application of effort into a concentrated area of the system and you get larger payoffs from it. This pattern relies on constraints and threshold effects within the system.

A threshold effect is like a tipping point. There’s some point in the system that you need to reach before you see any payoff. This is something that happens for example when you launch a new product. It can take a lot of effort to get to the threshold/tipping point of awareness about that product but once you’re there things start to become easier and you just need enough effort to maintain the effect.

The threshold effect can be understood through the concept of inertia from mechanics where it takes a lot of initial effort to get an object moving, but once it has momentum, you just need enough force to counteract friction.

Another application of constraints and threshold effects is when you go after a small market and seek to dominate it before trying to take on a bigger market. From a strategic point it is a lot easier to dominate a smaller market fully than to claim an equal slice from a bigger market.

A perfect example of the threshold and focused effort patterns is the strategy that Tesla Motors has used to take on the giants of the automotive industry. In a recent article in FastCompany.com, Tesla Motors founder Elon Musk stated that his plan for introducing the electric car was really simple:

1. Build sports car
2. Use that money to build an affordable car
3. Use that money to build an even more affordable car
4. While doing above, also provide zero-emission electric-power generation options

One of the investors in Tesla explained in an answer on Quora why they chose to first build the very expensive Roadster (the sports car) then the Model S (the affordable car) and the plans for what they’re calling the Bluestar (the more affordable version). The reason for this is that you can’t penetrate the automobile market from the bottom up, it has to be top-down.

The Roadster provided Tesla with many benefits they wouldn’t otherwise have if they started with an affordable car. They got money from sales to be able to sustain themselves, they created a brand image, they showed the world they could build a high-performance electric vehicle, and they learned a lot from the engineering challenges, lessons they can now apply towards making more affordable electric cars for everyone.

Part 2 – Building Assets

Now that we covered the patterns you can exploit in a system to gain leverage, let’s cover the other side of the coin which covers the assets, competencies and advantages that allow you to exploit those patterns.

There are three types of assets you may innately have or can build:

  1. Positions
  2. Resources
  3. Competencies.

1. Positions are a type of asset that has to do with either a physical location in space or a more abstract perceptual position in the mind. The first one is obvious. In real estate for example you know that the key is location, location, location. Where you are located with respect to people’s movement patterns can make you or break you. In war/conflict certain positions are more advantageous than others and thus become mission critical to get and to hold. Good examples are bridges, hills, trenches, etc.

The second one is more abstract in that it represents a position in your mind (or as Al Ries and Jack Trout in their seminal work Positioning – The battle for your mind call it cherchez le creneau). The central idea is that a product, a service or a company should occupy a very well-defined and simple concept in the consumer’s mind. For example when you’re thinking about a refreshing drink, Coca Cola wants you to think of Coke.

One of the key insights in the book is that you should not try to compete with a position that’s already been taken by a competitor but rather try and create a new position and place yourself, your company or your product there as the sole owner. You do this by inventing a category and becoming the only provider of products/services in that category.

Tim Ferriss (of the 4-Hour Work Week fame) didn’t want another book on “career advice” so he created a new category called “lifestyle design” and put his book as the first one there.

This doesn’t just work for brands and companies, it’s the same for people too. In a company for example you might be the “go-to guy” whom everyone calls in a crisis, or you may be the “recluse but brilliant scientist” whom nobody has met but everyone knows about. This position can make or break a career and should be chosen very carefully otherwise people will just slap some disadvantageous label onl you that will hinder your career.

There’s a lot that can be written when it comes to perceptual positions. I may do a full post on this in the future. Suffice it to say that how you position yourself will greatly affect your success in life.

2. Resources are a type of asset that represents concepts like money, connections, physical things like real estate, factories, farms, machinery, weapons, personal things like looks, height, weight, age, etc. They can be created, acquired or innate.

This is the most common type of leverage you have or can build because it’s really obvious when used. If you’re astute you will notice that resources can be used directly on one of the three patterns mentioned above (anticipation, pivotal points and focused effort) If you’re not using some kind of advantage to get ahead in life or in your career, you’re really being left behind and missing out. All those successful people you see have some kind of secret advantage they’re using, connections, physical features (height, weight, looks), competencies, etc.

3. Competencies are assets that reflect things you or your company are good at or have expertise in. They represent strengths but they’re slightly different than resources, although they can be called resources and just like resources they can be built (like skills), acquired (like experts) or innate (like talent)

A really good example of the use of competencies to exploit an advantage is the case of IBM getting out of the PC market and focusing on providing their customers tailored information processing solutions to their problems under the IBM brand name. They sold the PC business to Lenovo and moved their entire business into IT consulting knowing that they had the necessary competencies in place both software and hardware. That allowed IBM to keep growing while the PC business commoditized.

Putting it all together.

Competencies, resources and positions are at the mercy of the market forces of supply and demand as well as the three patterns we discussed above in part 1. They dictate whether or not the assets you have can be used as leverage.

The patterns of leverage are vast as you can see. I hope with the above I’ve been able to make a dent in this subject. Please note that this post doesn’t discuss the morality of how you use leverage. As with any power tool it can be used for both good and bad. My goal here is to simply make you aware of the various patterns of strategic thinking that we humans use to reach our goals or solve problems and hopefully help you by structuring it into something more usable.

Note: A lot of the ideas in this post came from Richard Rumelt’s book Good Strategy, Bad Strategy

Review of Little Bets – A Book on Experimental Innovation and Creative Thinking

LIttle Bets by Peter SimsLittle Bets by Peter Sims is a fantastic little book about experimental innovation and creative thinking. The subtitle actually reads:  “How Breakthrough Ideas Emerge from Small Discoveries” I put this book off for over a year before finally tackling it and I immediately realized my mistake.

The book is a collection of ideas on everything from launching new products, design, to creative warfare. There are a lot of little nuggets in this book and I’ll do my best to bring them to the surface. But first a personal story.

I work as a business intelligence analyst/developer, which is really a fancy term for analyzing business data, looking for patterns and then designing dashboard reports for business users such as VPs, directors, managers, to consume the data and make better decisions.

My most recent project required that I build a data mart for tracking our sales pipeline leads better. A data mart is business data that is in a very specific format which makes it easier to query and build reports on.

I started the project with a lot of enthusiasm and energy, after all you don’t get a chance to build something from the ground up very often. I had high aspirations and high standards for this project so I set out to build an all-encompassing data mart that could be used for multiple purposes. I had several things going in my favor:

  1. I was very familiar with the data. I’d been working with it for several years
  2. I had already built a similar project for tracking sales and it was working great
  3. I had began reading the definitive guide to building data marts and  I couldn’t wait to try it.
  4. I had built plenty of multi-purpose reports before
  5.  had met with the VP in charge of the project and had a very good idea of the requirements, down to specific deliverables.

Two weeks into the project, I was still trying to come up with a high-level concept for the way the data was laid out that would accomplish all my goals at once. I wanted the design to be perfect right from the start and I wanted everything to flow from there.

In other words I was stuck!

I battled with this for another two weeks before giving up my high-level, all encompassing concept and actually ended up designing a few specific reports to answer specific question. The project ran way over the allotted time and I was somewhat dissatisfied with the results. Why didn’t it work? The answers became clear to me once I started reading Little Bets

Thinking With Prototypes and Iterating

The book starts out with a bold principle: “Genius ideas never spring into people’s minds fully formed. They emerge through a process of discovery and experimentation” It’s a bold principle because traditionally we’ve been trained to think in terms of top-down design; first come up with the idea in our mind then take action. That’s exactly what I was trying to do! I was trying to come up with a brilliant design first and then build the reports after.

Only then did it dawn on me that the process I followed to build the project on sales (item #2 above) was the same process of creative discovery this book was talking about. It was only after building several purpose-specific reports all based on the same, common data set that I was able to see the patterns and be able to abstract the data to an all-encompassing, all purpose data mart.

One of the core ideas in the book is the idea of thinking with prototypes. Those of us in the software engineering field know this principle very well. It’s at the heart of the Agile Method. Prototypes allow you to explore various ideas and various designs until the final product emerges. But there’s a caveat.  As John Lasseter of Pixar says “We don’t actually finish our films, we release them” The agile method says “release early and often”

I think that’s very telling of the mindset that has made Pixar a million-dollar successful animated films empire. It’s very important to actually put stuff out there and get feedback especially from the actual users of your product. That’s the key to the iterative method which stands in sharp contrast to the old-school “waterfall” planning method. This method which grew out of the way manufacturing and construction industries handled their projects, has turned out to be disastrous when applied to software engineering.

Little Bets and Entrepreneurship

The book expands on this principle by taking it into the world of entrepreneurship and product ideas. It states that most successful entrepreneurs don’t begin with brilliant ideas, they discover them through a process of experimenting and taking little bets. According to Sims, little bets are “concrete actions taken to discover, develop and test ideas that are achievable.”

Here are a sampling of nuggets from the book:

“Accept uncertainty, that you cannot reliably predict which ideas will work and which will fail and most of them will fail, so you have to experiment and try things in the real world. Approach problems in non-linear fashion using little bets, what David Galenson has dubbed “experimental innovation. Experimental innovators must be persistent and willing to accept failure and setbacks in the pursuit of their goals. We don’t know what we don’t know so instead of trying to develop elaborate plans to predict success, you should do things in order to discover what you should do.”

In my case, I should have begun by building the specific reports and fine-tuning them. As one of my friends says “humans are tinkerers.” That’s why when starting a new project (he’s a software engineer) he always starts with the smallest possible set of features that works. If he’s building an app, he will have a crappy UI, and crappy code as long as the app successfully builds and runs even if it’s just to display a message like “hello world”

How entrepreneurs think was the subject of a study and paper by professor Saras Sarasvathi. In 1997 she set out on a journey across 17 US states to meet with 30 founders of companies ranging in size from $200 millions to $6.5 billion dollars and to find out what makes entrepreneurs entrepreneurial. She came up with a theory she calls “effectual reasoning” which differs quite a bit from the causal reasoning we’re all taught in school.

Here’s how she puts it: “Effectual reasoning does not begin with a specific goal. Instead it begins with a given set of means and allows the goals to emerge contingently over time.” According to Sarasvathi, all the entrepreneurs she studied began with three categories of means: (1) Who they are – their traits, skills, values (2) What they know – their education, training, expertise and (3) Whom they know – their social/professional network

“Effectual reasoning is all about execution. Plans are made and unmade, revised and recast through action and interaction with others” These entrepreneurs don’t like to do traditional market research such as focus groups, targeting, etc. Instead they prefer to talk to customers and find out what they want. They take product ideas and half-baked prototypes directly to customers to see if the idea works.

Effectual reasoning has three core principles according to Sarasvathi.:

  1. The Affordable Loss Principle. While managers try to analyze the market and choose targeted segments to launch new products, entrepreneurs try to reach the market quickly with as few resources (time, effort, money)as possible, as well as taking a small enough bet that you can learn from it without it costing you the your entire savings. If you have ever seen the show Shark Tank, one of the saddest moments in the show is when a prospective entrepreneur has spent years and years and probably drained their savings to build a product only to see it rejected because they never took it to market.
  2. The Strategic Partnership Principle. The successful entrepreneurs in this study all prefer to build strategic partnerships rather than focus on competitive analysis. They start these partnerships as soon as they have customers, in fact customers are considered partners right from the beginning.
  3. The Leveraging Contingencies Principle. This is the ready-fire-aim approach to building products. The ability to take an unexpected turn of events into a profitable business is at the heart of entrepreneurship. Many great companies (Google, Microsoft, Apple, etc)  were built from leveraging contingencies.

The Growth Mindset

What I found both interesting and fascinating is the fact that Little Bets has an entire chapter dedicated to one of my favorite books about psychology. This other book is called Mindset: The New Psychology of Success by Carol Dweck. Dweck is one of my favorite researchers in psychology today. Her book tackles the theory of the growth mindset vs. the fixed mindset while applying it to a variety of fields from intelligence and learning to self-help.

She found while studying students that some of them would prevail in the face of challenges while others would falter, give up or not even try. Going deeper, she found that the key was their attitude towards learning. The students who were able to overcome difficulties thought about learning from the perspective of growing their abilities and intelligence while the ones who gave up or didn’t try thought about learning as having a fixed intelligence and either proving or disproving their own smartness.

This is one of the things I’ve always battled with, having been praised my whole life about my intelligence and innate talents and abilities. I’ve always wanted to grasp things immediately and understand them fully. Failure to do so, meant I was not smart so I’d avoid those situations as much as I could. It wasn’t until I read Carol Dweck’s book that it started to make sense. Little Bets looks at this idea from the perspective of how we view failure. Here’s a little table exploring the two mindsets.

Fixed Mindset Growth Mindset
Fixed intelligence/Innate talent Growing intelligence/Learnable Talents.
Must prove myself to be smart or else be seen as stupid Failure means feedback. There’s no need to prove myself because I’m learning
Confirming abilities Expanding abilities
Take no risks unless you’re sure you will succeed Take risks because whether I fail or succeed I will still learn
Desire to look smart Desire to learn
Avoids challenges Embraces challenges
Gives up easily Resists in the face of setbacks
Effort is fruitless Effort is key to mastery
Ignore useful negative feedback Learn from criticism
Feel threatened by other’s success/achievement Find lessons and inspiration in other’s success/achievement
Plateau early Reach higher and higher levels of success

How do you change your mindset? Carol Dweck says: “A person’s mindset can strongly be influenced by what they think is most important ability or effort” Once you become aware of your mindset, you can start to influence it by paying more attention to the effort you put into it vs. thinking about your ability to do it. It is very important to praise effort in children early on rather than ability because that’s how you build the growth mindset.

This is just a small sampling of the goodies in this book and highly recommend you read it.

The Dangers of Optimization

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil” -Donald Knuth

In the field of Computer Science (CS) optimization is the process of changing the algorithm (the logic) of a program in order to improve its efficiency (i.e. make it run faster or consume fewer resources.) In order to do this, certain assumptions need to be put in place and my argument is that these assumptions make optimization dangerous and not just in the context of programming.

First of all in order to optimize a process you need to understand it very well. You need to know how it behaves in different circumstances or under various boundary conditions. If we were to look at an organically (necessity) grown business process that we want to optimize, it’s usually necessary to see where the inefficiencies or bottlenecks are in the process and remove them.

Second, when you optimize you may need to restrict boundaries, make certain assumptions, use special cases, tricks and complex trade-offs which will achieve the required result but at the expense of potentially over-complicating a process or over-specializing it. This causes a loss of agility in the long run or over-adaptation (as may be the case in over-optimized diets).

Let’s use a simple example from manufacturing to explain this. Let’s assume that you have a retool-able machine that produces widgets at 95% defect rate. This means that 5 out of every 100 widgets are defective. When you’re looking to optimize the defect rate, you’re looking to produce fewer defective widgets without slowing down the machine or the manufacturing process. Suppose also that you know the process of making this widget very well, since you make millions of them every year.

Since you know the process well you know that there are just a few ways to optimize the defect rate, you can for example utilize the machine better by redesigning the overall process or you can install much more specialized machines that make just this one widget. Now you have an optimized defect rate but it has come at the expense of over-specialization and a huge loss of flexibility or agility. Imagine what can happen after you’ve replaced your retoolable machines with specialized machines and the market changes to where it no longer needs those widgets. You’re practically out of business.

The danger is that not all processes are fully known. Even in a precise field such as computer science there’s always some level of unknown, certain unexpected conditions or special circumstances where the program will fail. If you were to optimize the process without knowing all these cases you run the risk of having an incorrect program that no longer solves the original problem.

Nowhere is this more prevalent than in digital marketing. First let’s get one obvious thing out of the way. SEO or search engine optimization is not really optimization per se. You’re not optimizing your website to work better or faster, you’re adapting it to fit the search engine’s constraints.

The whole idea behind the concept of targeting is really about optimization. When you’re trying to target a certain segment of your audience, you’re trying to optimize the marketing dollars so you increase the efficiency. After all why would you want to spend money on leads that will not respond to your offer?

There is nothing wrong with targeting (although Roy H. Williams aka The Wizard of Ads would beg to differ) The problems arise when you start to over-optimize while assuming that you really understand your audience and what offers they actually respond to. The truth is you really don’t, so all this optimization will really hurt sales. There are many other things in a business you can optimize and improve, such as your close ratio.

A good example of this is a recent story I read in Roy H. William’s Monday Morning Memo (a highly recommended business newsletter by the way) The story is about two lawyers who take different approaches to marketing. One believes in targeting, and converts about 10% of inquiries while the other believes in casting a wider net via radio advertising and ends up converting 60% of inquiries proving once again that marketing is primarily about the messaging and the offer.

The 12 Tenets of Persuasion

The art of dealmaking is much more than just persuasion, negotiation techniques or networking skills. You have to become a master communicator if you’re going to have any chance to sell your ideas and move up in life. The best persuaders are the best communicators.

The following is a loosely based set of beliefs, mindsets, frames/frameworks and mental models. My preferred definition is mental model or mental frame culled from various sources.

1. High Value Assumption. The idea behind high value assumption is that you always assume and speak from a high value frame for yourself. That is, regardless of whether you have experience, knowledge or networks to rely on, you still value yourself highly and you aim to increase or keep that higher value. This means that you never sink to low-value behaviors such as begging, supplicating, giving away free compliance, pushing, etc.

This stance allows you to keep your self-respect and walk away from any situation even if you don’t get what you want because again the aim is to preserve the high-value stance. Also remember that in general people want to be led so lead them but don’t try to control them. If they’re unwilling to come along for the ride, then you shouldn’t try to force them. That would be low-value behavior.

2. Mutual Value Exchange/Escalation. The idea behind mutual value escalation is that in any situation or interaction you’re always trying to increase the value that both parties get. This is a play on the meme of win-win. However MVE takes win-win and expands the realm of possibilities of winning. You may think you know what the other side would consider a win but you will be wrong most of the time since you’re judging it based on your own criteria of winning. When you look at it from the perspective of making sure they get value out of it while you’re also getting value, it makes it easier to negotiate.

As applied to persuasion, it states that whenever we attempt to persuade, we must do so in a way that enriches and expands the mental models of others and gives them choice rather than takes away choice. There are many persuasion techniques out there that are considered unethical that work by restricting choices to the ones that you want them to take. In fact my definition of unethical persuasion is one where you’re striving to restrict people’s choices and ethical persuasion is one where you strive to augment their choices and enrich their mental models. The use of choice-restriction techniques in ethical persuasion (such as the take-away, scarcity, etc) is only done as a way to jar people’s current restrictive models and force them to re-evaluate their options.

3. The person with the most flexibility and the most options exercises the most influence. This is one of the core tenets of cybernetics and systems theory and is also known as the law of requisite variety. As applied to persuasion, it states that the person who has the most choices of response is the one who’s going to end up leading the interaction. In chess, whoever can think one step further than their opponent usually wins. This is another facet of the law of options.

4. People respond to you based on their own mental models. This basically states that when you talk to people and you try to persuade them, you need to understand what model or frame are they using to communicate with you on that specific context and use that to either pace them or lead them. The basic understanding is that until you know for sure how the other person thinks, you can’t assume that they think exactly like you do. This is the most common mistake people make when communicating or persuading. We must respect the other person’s mental model and not try to change it to be like ours. This is the second most common mistake we make while communicating.

Rapport is achieved when we step inside the other person’s mental model, even if for a few seconds, and we see things from their perspective. We can then choose to pace it and then lead them or we can choose to ignore it, assume the higher value and qualify them. It all depends on us, but the fact that we have options makes all the difference. (see the above law/model/belief) I call this technique “getting inside their head” When you don’t have rapport you will experience a lot of resistance.

5. Context determines meaning. While you can attach any meaning to any concept or behavior, more often than not, the context will determine the meaning of any behavior or communication.

6. The meaning of your communication depends on the response that you get. This tenet derives from the rule of flexibility. It puts the responsibility for communicating the right thing on you and now on the other person understanding what you say. Things mean whatever the other person thinks they mean not what you intended it to mean. This will allow you to change your behavior and be flexible in your communication until you get the desired response. (see tenet 3)  People haphazardly gain meaning from experiences and information so why can’t I be the one who gives them the meaning ahead of time?

7. The one who sets the frame for the communication controls the outcome of the communication. This tenet basically states that how you frame a particular behavior controls or directs how people will respond to it. A frame is essentially a perspective, a way how you see things. If you want to be the one driving the persuasion or communication, you should be the one who sets the frame. If the other person however happens to set the frame that you want, there’s no need to one-up them, so just follow along. Remember the principle of variety and choices in response. Here’s a good example of setting the frame:

  • “The sun has a beautiful red color to it as it’s setting tonight. (frame) Let’s take a walk on the beach”
  • “It’s going to be too dark when we get there (new frame- Dark is not good)”
  • Seductive voice ‘Well that will be nice. That way no one can see what I’m going to do to you once we get there’ (reframe- Dark is good)

One of the most important,  basic, core skills in persuasion is the ability to reframe a situation to mean anything we want in order to drive the persuasion in the direction that we want.

8. Objections indicate lack of perceived value. In almost any persuasion situation, the primary reason why people object to your “sales pitch” is the fact that you didn’t build enough value for your product or idea, and by value I mean value that is specific to their situation.

9. Value = Believable Result. Value is a lot of things, and we’ll get into the specifics of it later, however at the core, you’re not selling a product or yourself, you’re selling the most valued results that the product (or yourself) provides someone.

10. There is no failure only feedback. This means that all the responses you get are purely feedback regardless of whether they are the intended result or the unintended result. In fact even if you get what you intended, you should still see it as feedback rather than “success” vs. “failure” This same principle can be applied to situations where you think you may say something stupid which will hold you back. There are no situations that you can’t recover from.

11. Time is the most important consideration to a persuader. (credit Doc Sulo) The most common belief amongst people out there is that time is far more important than money because you can always get more money but you can’t get more time. So as a persuader you should try to minimize the time spent but that doesn’t mean you should strive for quick results. It does mean however that you should try and gather as much information as you can, plan and strategize rather than try persistently. This means you don’t want to get a quick kill.

12. A great persuader will always leverage time spent persuading into maximum benefits over time. (credit Doc Sulo) What this means is that when you spend time to gather information and persuade someone, you might as well leverage that time into long-term benefits. This means that you should focus on long-term persuasion vs. short-term persuasion. In this book we will only discuss long-term techniques. Time is important but you have to plan things out in advance from the beginning all the way to the end and using strategy to use your time in the most beneficial way. Tim Ferriss always talks about making friends and creating relationships with people who will further both of your options in the future, thus be mutually valuable. He will cut all the “dead weight” of relationships and partnerships that are either suffocating your value or no longer bringing in any new value.

How I Cured My Shyness – An Introduction to Refactored Thinking

I haven’t always been the social, chatty guy that I appear to be today. In fact as many of my high school friends would tell you, I was pretty awkward and shy in social situations. This never affected my life in college, as I had no problem talking to the group of friends I hung out with, after all we were all nerds who talked about programming and philosophy.

After college, we all went our separate ways and so there was no longer a group of people who I could socialize with. I found myself feeling pretty isolated, I mean I couldn’t even hold a simple conversation with a stranger and talking to girls in loud bars was completely out of the question.

I have studied psychology my entire life. In high school I was somewhat of an amateur psychologist whereby I tried to understand people’s motivations for doing things with the limited models I had available. I have also been heavily involved in self-improvement. I’ve tried everything from hypnosis to NLP, to positive affirmations.

Some things worked sometimes, but nothing really stuck and my “shyness” continued to plague me. It wasn’t until about 4 years ago when something remarkable happened.

I was reading yet another book about how to be more social, in fact I think I had already read the book once. My state of mind on this day was one of frustration. How can it be, I thought, that despite all these things I’ve studied and all these things I’ve done, I still have no idea how to permanently cure my shyness?

As I was reading the book again, a sentence in a paragraph jumped out at me and gripped my temples in a big jolt of excitement! The answer I was seeking, the secret that permanently cured my shyness was right there in front of me. I had missed it!

So what happened?

Before I tell you what happened allow me to set the stage.

I had been exposed to the idea of mental models before; they were called beliefs. Beliefs are essentially generalizations (mental scripts) that we believe without question. They not only lead to automatic behavior but they also tend to define your reality.

For example when you reach a conclusion on your own about something or someone, you’re building a mental model (a belief) about that thing. Now that the “hard work” of thinking is done the brain, ever the effort optimizer, will go on and use that conclusion as the truth in guiding your thoughts and actions.

Ever since I remember I have been labeled as “smart.” I was a little ahead of the other kids when it came to reading and comprehension and I mostly coasted through school getting good grades with minimal effort. My peers always asked me for help with difficult subjects such as math and physics. In fact my nickname was “professor” since I always tried to teach new things to my friends or to explain things to them.

This is all fine and dandy but without realizing it, I had developed a mental barrier to information which I didn’t quite understand. In fact, I was pretty adamant about not being exposed to new information which I didn’t understand (or else run the risk of being labeled stupid)

Carol Dweck explains this phenomenon at length in her book Mindset – The New Psychology of Success She calls it a “fixed” mindset where the fundamental belief is that you cannot learn and grow and get better. I read this book much later after this initial epiphany.

For me as it turned out, protecting my “smart” label had taken on a life of its own inside my head and as I discovered, had been the core driver behind my apparent shyness.

The paragraph that cured my shyness

Getting back to the story, the book I was reading was an eBook by Dr. Georges Sabongui called The Art of Social Networking I don’t think it’s available anymore. The paragraph was this:

“….and if I had no idea about a particular topic, I wouldn’t turn off, I
would turn on. I would get excited about the idea that this person I
was talking to could teach me something I didn’t know about and
broaden my horizons…”

By itself the paragraph is pretty harmless and powerless, but when feel as frustrated with everything as I was and when you read something that fits your specific need so perfectly that it feels as if it was custom tailored to you, the experience is nothing short of an epiphany!

Some of you may recognize this as plain old reframing, and it IS a reframe, but it’s more than that to me. It’s one of maybe dozens of different reframes that would have never worked for that particular situation. That’s why I call it thought refactoring.

In software engineering the concept of refactoring means that you take existing code that works, and you rewrite it in a way that is cleaner, simpler, more readable and more organized without losing the original functionality. It’s a way to go back and rethink about the problem you were trying to solve in a different way.

Thought refactoring works the same way, except that you don’t need to worry about keeping the same functionality as long as the original problem is still solved. It can be as simple as writing down your thoughts and then rewriting them in a more concise manner and as complicated as debugging your own brain.

While I don’t buy into the brain-as-a-computer metaphor, since it’s been proven wrong by modern cognitive science, the idea of going in and replacing “buggy code” in your mind in the exact location where it’s needed, still appeals to me. Ramit Sethi calls it replacing invisible scripts.

What happened next

That same afternoon after my big a-ha I went to a meetup event with a big group of strangers some of whom much smarter than me, and I found myself having absolutely no problem stating my own thoughts and debating (it was a discussion group) In fact I felt so liberated that I went to another meetup right after that one with a guy I met for the first time at the event and had no problem conversing during the train ride. It’s been awesome ever since!

Becoming a Master Dealmaker – Two Meta Models of Persuasion

I have a passion for persuasion. I think that everyone should know how to persuade ethically and effectively. This obsession has led me to study psychology, strategy, marketing, cognitive science, and anything related to persuasion and influence that I could get my hands on.

I have observed that there are two main models of persuasion that are in use today. These are more like meta models of persuasion since they encompass many little models under them.

The needs based model.

As the title suggests, the needs based model relies on us understanding the various needs of another person and designing our products and services to fulfill those needs. While seemingly simple in surface, the needs based model is the core behind almost any form of marketing and advertising today.

The basic idea behind this model is that people have wants and needs such as the Maslow hierarchy of needs and we are motivated to fulfill those needs. Marketers then direct their attention towards linking their product or service to a particular need.

Maslow’s list includes:

  1. Physiological needs, such as food, water, bodily comforts, and other individual biological needs.
  2. Safety needs, such as shelter and protection, free from worry about things like money, health, etc.
  3. Love/belonging needs, such as friendships, relationships, acceptance, family, intimacy, camaraderie, etc.
  4. Esteem needs, such as self-esteem, confidence, achievement, respect from others but also status, glory, reputation/fame, etc.
  5. Self-Actualization needs such as morality, creativity, spontaneity, lack of prejudice, effortlessness, achievement, problem solving, playfulness, etc.

Along with Maslow’s original list of needs, there other needs that are slightly more “negative” or darker/sinister in nature, far more secretive and powerful. They too are part of the human nature and cannot be ignored. In fact one book I read called them appropriately “hidden addictions” This list includes:

  • The 7 “deadly” sins/vices: Lust, Greed, Gluttony, Sloth, Wrath, Envy, Pride. All of them very powerful and very easily taken advantage of
  • Vanity
  • The need to be right instead of wrong
  • The need for a scapegoat
  • The need to know secrets that nobody else knows
  • The need for power (or a sense of power)
  • The desire to win (or not to lose)
  • Etc.

The basis of the needs based model is that you first determine what the other person’s most pressing need is and then you frame your offer in a way that gives them the hope that you can fulfill that need, without anyone directly stating the need. This is how scam artists usually take advantage of people.

The biases/shortcuts based model.

The shortcut/biases based model takes advantage of the brain’s inherent desire to preserve as much energy as possible and use heuristics to make decisions rather than think harder. The list of cognitive biases is HUGE!

By presenting an offer in ways that take advantage of these biases (such as for example the status quo bias or the confirmation bias) you can get someone to make a quick decision in your favor.

For example we tend to remember much better things in the beginning and things in the end and not so much the things in the middle, so if for example you start off the year working hard, you slack off in the middle but then you pick it up again toward the end of the year (when reviews are usually done) people will tend to remember more what you did last and thus give you a higher score.

Another very important example is the status quo bias where you tend to go with the default choice rather than weigh all the choices evenly. “Evil” websites and software developers for example can take advantage of this by leaving checked off options where you are charged a recurring fee, or where you inadvertently install programs you never intend to use.

The core distinction among the two models is customizability.

When you’re creating a product to target a very specific need or want, you HAVE to customize your pitch for the appropriate audience or else you won’t connect with them. (By the way this is also how you gain rapport with people….more on that on another post)

However when you’re using biases and shortcuts, since we all have pretty much the same biases, you can keep your pitch constant regardless of the audience you’re presenting to. This is the main idea behind a book called Pitch Anything by Oren Klaff. He focuses primarily on how you can frame your product in such a way that it will appeal to any audience without the need to vary it based on needs.

The most effective persuasion techniques combine both models for the biggest bang for your buck.