Why Amateur Economists Matter

Amateur economists matter because they see properties of the economy that professional economists cannot see or do not wish to see. Academic economists have a tunnel vision problem, they avoid economic issues where a mathematical model does not fit - and some of the most important properties of the economy do not obey mathematical equations. The economic incentives in an academic economist's career encourage him to focus excessively on models.
What about journalists who write about economics or corporate governance? Journalists have their own groupthink, in terms of what questions are worth asking and what's worth writing about. There are a few excellent contrarians (Gillian Tett and John Kay) who are not afraid to point out when economists (and financial industry experts and corporate executives) are flat out wrong. That said, the mainstream media has substantial biases (everyone does) - so there are vitally important characteristics of the economy they do not wish to ask questions about.

Amateur economists may not get respect or prestige, but their insights can matter. Here are some properties of the economy where an amateur economist can provide better insight than an academic economist:

* Corporate executives make poor decisions to a greater extent than economists realize. The economy is more ossified and sclerotic as a result. Reasons include socio-economic discrimination, tunnel vision, looking only at quarterly financial numbers and not making the effort to examine the operations behind the numbers, not taking a long-term view, and lack of supervision by institutional shareholders.

* Corporate executives have no idea how to manage technology workers. They cannot be technology experts. But they also make no effort to manage the political/economic incentives that drive technology workers' behaviour. (Technology workers don't want worker productivity growth if they feel it threatens their career.)

* Corporate executives' preference is to lock innovators out of the building, because their socio-economic status is usually low. Corporate executives make decisions based upon socio-economic status, not based upon the merit of an idea or product. (In a nutshell, I'm saying that corporate executives are shallow, and for that reason lack the willingness or intellectual flexibility to tackle long-term cost-efficiency problems.)

* Socio-economic discrimination is very prominent in today's economy. Not only does this damage the economy, the effect on individuals can be demeaning, and it discourages innovation. The higher the rate of socio-economic discrimination, the more sclerotic the economy. The result is less innovation, lower economic potential, lower worker productivity, and lower total factor productivity.

* Nobody cares about worker productivity. (Yes, this includes the CFO.) Hence, impediments to worker productivity growth remain in place for decades - or longer.

* Nobody understands worker productivity. Why will worker productivity spurt up, and then stagnate for decades? Academic economists do not have a clue. Mathematical models can't explain that, you have to watch how people behave. But to understand worker productivity, you first must understand that nobody cares about worker productivity. You also have to understand what happens to people who try to solve such problems (bad things happen to them).

* Nobody cares about the long-term (look at what they do, not what they say). The most obvious examples are corporate executives, institutional shareholders, and taxpayers. A less obvious example are those who control financing of research. This is a huge factor in economies becoming stagnant and sclerotic.


* Everyone has a bias, usually specific to their choice of profession. This is because of economic/political incentives within a career path, and also old established habits of the profession. Everybody. Including me, including you. In general, this over the long term makes the economy more sclerotic.

* People will do what they have an economic incentive to do. Without the incentive, they won't do it. I know that sounds over the top, but the range of exceptions is strikingly narrow. Raising a child obviously falls within the range of exceptions, but not much else does. You think charity workers are an exception? Then read Gillian Tett's utterly appalling essay on anti-poaching efforts. A more important example is ...

* Nobody has an economic incentive to solve cost-efficiency problems in medical care and medical research. So nobody tries to do it. This is true even though the high cost of medical care damages a lot of people's lives. [ And this is precisely the sort of economic problem where academic economists are so useless, why drug prices are high is my own expertise ]. Health care executives are NOT trying to solve long-term cost-efficiency problems.

* Pharmaceutical CFOs are incompetent. If institutional shareholders were not wasting time sleeping under the desk and eating doughnuts, the CFOs would have been fired long ago. My previous blog post went into detail on this.


* THE ROLE OF THE MAINSTREAM MEDIA IN THE HIGH COST OF HEALTH-CARE:
The high cost of health care is a big problem, and institutional investors and the mainstream media make the problem worse. Here's why:
Health care executives are not motivated to solve long-term cost-efficiency problems. Hence, the high cost of health care. Obviously, this damages people's lives. You would think that gung-ho journalists would be digging deep for the scoop, going into the details. But for the most part, the mainstream media doesn't really care about the politics and economics inside the health care industry. Both the mainstream media and institutional investors play a very quiet and passive role in this area, this encourages health-care executives to continue to not make an effort.


* Technology workers are Luddites. More about this in a later blog post, but examples include: movie special effects, Javascript coders, and of course, pharmaceutical SAS programmers. Technology workers usually have an economic incentive to stop worker productivity growth.

* The economic interest of a technology worker is often aligned with the economic interest of a software vendor and NOT aligned with the economic interest of the shareholders of the company that actually cut's the technology worker's check. This is a problem across the American economy, but later blog posts will mention pharmaceutical SAS programmers: pharmaceutical SAS programmers have an incentive to protect the SAS programming language AT THE EXPENSE OF pharmaceutical shareholders.

* Technology companies do not want worker productivity growth. The more workers have to master the proprietary platform, the more powerful the software vendor is in negotiations with the customer corporation. (For example, in negotiations between SAS Institute and pharmaceutical executives, SAS representatives have total unquestioned power. The reason for this is that pharmaceutical CFOs have been incompetent in managing long-term cost-efficiency issues.)

* Chief Financial Officers are simply not very good at playing the long game. Perhaps the pharmaceutical industry is an extreme example, but this is a problem across the economy. (As you'll see in later blog posts, I generally assume that, regarding management of long-term cost efficiency problems, it's the CFO whose head should be on the chopping block. In any case, it has to be either the CEO or the CFO. If there are many people to blame, then no one is accountable.) For example, watching pharmaceutical CFOs spar with SAS Institute is like watching a chess grand-master spar with someone who couldn't beat a five-year old in checkers. Yes, pharmaceutical CFOs have been that incompetent on this long-term cost-efficiency issue. I do not exaggerate. My next blog post will explain.

* Short-term cost cutting is very different from solving long-term cost-efficiency problems. They are different problems that require different techniques. For example, layoffs do not usually raise long-term worker productivity. But here's a clue: you have to look at incentives. Do workers have an incentive to solve worker productivity problems? Do workers have an incentive to stop worker productivity growth and lock innovators out of the building? Institutional investors and the media do not make the distinction between short-term cost cutting and long-term cost-efficiencies, and if investors and the media do not make the distinction, do not be surprised if corporate executives also don't care.

* Worker productivity is NOT a measure of how hard-working or lazy a worker is. (It is a contributing factor, but there are other factors.) A surprising number of corporate executives don't get that. Measuring time at the water cooler is not a measure of worker productivity.


* Wall Street often does not care if corporations waste a lot of money. Investors do not take the long view, so it's okay to waste a lot of money as long as the company is raking in the cash on the revenue side. It's a bad idea for an institutional shareholder that claims to take the long view to do this, because the longer you wait, the worse it gets.

* Institutional shareholders do a lot of damage to the economy by what they don't do. They are very lazy thinkers, and they are not independent thinkers. Group-think is a big factor: one shareholder will not examine an issue because other shareholders are not. They make the economy much much more sclerotic. And considering the number of employees working at pension funds and the like, that is a huge amount of wasted brainpower. By far, institutional shareholders are the weakest link in the economy.

* The COMPETE/COOPERATE problem : when you have a compete/cooperate problem, you have an issue that can create a mal-functioning economy. You have a group of entities (institutions or individuals) that often need to compete against each other, but in some situations they need to cooperate in order to solve long-term problems. The most important example is institutional shareholders.

* The university sector matters as much as the corporate sector. Aside from actual scientific limits, the economic/political incentives in research careers can cause serious long-term economic problems, and delay innovation for centuries. Scientists often have a deep hostility to new ideas and contrarian researchers.

* I call this problem the "TWO OUTFIELDER PROBLEM": problems that fall at the fuzzy boundary between the public sector and the private sector can remain unsolved for decades. Like in baseball, a pop ball goes straight up the middle, you see the ball on the ground and two outfielders pointing the finger of blame at each other. A lot of applied computer science problems are like that.

* "THE OPAQUE/TRANSPARENT PROBLEM": If you are a small vested interest, and you want to protect your economic status, then secrecy is great. If you are in charge of an operation or department that outsiders struggle to understand, then you have that secrecy, and secrecy is power. Many vested interests scattered throughout the health-care industry use secrecy to stop cost-efficiency problems from being solved. And corporate executives also use opaqueness to justify their salaries, but a lot of people who play this game are upper-middle-class workers.

* Regulations have many reasons for existing. Some are there for good reasons, others are there to protect vested interests, or by accident. But if you are a full-time employee looking after your own career, never criticize a regulation. Talk as if all regulations are good and not to be questioned. A later post will cover this.


* Thomas Malthus was right. Now maybe some brilliant researcher will make some great discovery at the intersection of psychology and quantum mechanics ( you cannot collapse the Schrodinger wave without a life form, something that physicists really really don't want to talk about ), and economic potential will surge. But the implications may also not be good economic news, and in any case, physicists will do everything in their power to make sure that contrarian researchers are denied financing. At the moment, it does look like Thomas Malthus was right, and academic economists are not comfortable recognizing that reality. North America will do OK for this century, but expect per-capita GDP in western Europe to stagnate and then begin to decline sometime this century. And the next century? Buy a gun and lock the door.


* "Crack-The-Whip" worker productivity growth and structural worker productivity growth are not the same. I suspect most of the worker productivity growth over the past several years is ctw-worker productivity growth ("ctw" stands for "crack-the-whip"). This involves, for example, dismissing lazy workers, a necessary task. It can also involve making hard-working workers work harder (via fear, for example). It's a result of the shift in the balance of power between capital and labour. But this is a cyclical effect, it's not where rising living standards come from in the long run. Rising living standards come from long-term structural worker productivity growth. This is from careful implementation of technology, or changes in business processes. In my experience (certainly in the pharmaceutical industry), corporate executives deserve no credit at all for structural worker productivity growth. They have great respect for people with high social status and the right political connections, and no respect at all for people of a lower economic class who solve structural worker productivity problems. Executives deserve no credit for structural worker productivity growth because they are shallow and lack imagination.

* There are large parts of the American economy where the "FREE-MARKET CAPITALISM MODEL" is not at all accurate. A later post will elaborate, but examples include: the pharmaceutical industry, the entire health-care industry, the market for corporate executives' salaries, government(of course), university research, sports, education. The higher the rate of socio-economic discrimination, the more likely that the free-market model does not explain that part of the economy. To make the problem worse, some people treat the free-market model as a theory as explaining how the economy works, but others treat the free-market model as an ideology.

* My last bullet-point is both tongue-in-cheek and dead serious. You can gain deep insights into economics by insulting everybody. Of course, taking the most cynical view of human nature will not give a 100% accurate understanding of the economy. But it comes surprisingly close, certainly more accurate than all the mathematical toys that academic economists play with. To be a good economist, don't go out of your way to insult people, but don't be afraid to insult people. Again, Tett's article on anti-poaching is a useful example showing how ugly human nature can be (we're talking about charity workers here!). In order to deal effectively with the high cost of healthcare, you have to reveal truths that people in the healthcare industry do not want under the sunlight. Someone who tackles the root causes of the high cost of healthcare will have few friends and many enemies.


OK, that's a pretty big list of insights on how the economy REALLY works. But each bullet-point above shows a dimension of the economy where an insightful amateur economist can provide valuable insights that an academic economist cannot or will not provide.

Fire Derica Rice

Derica Rice, the Chief Financial Officer of Eli Lilly should be fired. He's doing a lousy job and he is certainly not worth anywhere near what he is paid. The reason why he keeps his job is because institutional shareholders don't do their homework. I worked at Eli Lilly in 2007-2008. Eli Lilly has a deeply entrenched cultural problem: a lack of political will for solving long-term cost-efficiency problems. Eli Lilly has a lot of structural cost-efficiency problems and worker productivity problems that have remained unsolved for decades. These problems, since they are cultural, cut across all operations and departments (my particular expertise is computer programming services, where pharmaceutical shareholders consistently get ripped off). Derica Rice has done almost nothing to deal with these long-term cost-efficiency problems. He is not competent, he is not really trying, he should be fired.

The pharmaceutical industry has a deep cultural problem: a total lack of political will to solve long-term cost-efficiency problems and worker productivity problems. To understand this cultural problem, consider this example: the worker productivity problem for providing diverse and complex statistical tables to statisticians and medical researchers. (The SAS programming language has serious worker productivity problems for several areas, including tables.) Each statistical table request typically requires over 400 lines of code (hours of work), but an outsider with common sense would see that most requests should not require more than 20 lines of code (20 minutes of work). This is a waste of pharmaceutical shareholder's money. This problem has been around since the 1980s, maybe earlier. This is a problem that could have and should have been solved by 1995. Why wasn't it? Because IT'S NOT A COMPUTER SCIENCE PROBLEM - IT IS A CULTURAL PROBLEM. First, pharmaceutical SAS programmers have an economic incentive to NOT solve worker productivity problems. Second, the pharmaceutical industry is deeply hostile to innovators and to new ideas in general. Both pharmaceutical SAS programmers and pharmaceutical CFOs prefer to keep innovators locked out of the building, but for different reasons. Pharmaceutical SAS programmers (especially senior SAS managers at CROs) want to protect their economic status by stopping worker productivity growth. Pharmaceutical CFOs have a different reason.

CFOs only want solutions and innovations that come from people like them, other members of the corporate elite. [ This is called SOCIOECONOMIC-DISCRIMINATION, and I'll write in other economic posts how it damages the American economy.] In general, corporate executives have deep contempt for the poor and middle class, a problem that is becoming worse as the American economy becomes more ossified. In this context, it means they find it hard to believe that worthy solutions come from an innovator who is not like them. So when corporate executives receive letters from innovators from a much lower economic class, they instinctively toss such letters in the trash.
In a nutshell:
Pharmaceutical SAS programmers are protecting their own economic interests.
Pharmaceutical Chief Financial Officers are shallow people.


So that is why this worker productivity problem remained unsolved for decades - because it's not actually a computer science problem, it's a cultural problem. [And as I'll argue in later posts, it's a cultural problem not only across the pharmaceutical industry, but in the broader healthcare industry - and this damages a lot of people's lives.]
What happens when an innovator announces a solution? You still have the same cultural problem. Pharmaceutical SAS programmers (and especially CROs) will do everything in their power to keep the innovator locked out of the building. And pharmaceutical executives continue to toss letters from the poor and middle class in the trash.

I sent Rice, and the CFOs of Biogen and Pfizer, at least two letters each explaining the situation. I called Rice twice, but of course the receptionist's job is to protect the corporate executives from the scumbags of the lower economic classes. [Disclosure: I have not worked at Pfizer or Biogen, but this is a problem in the entire drug industry.] I sent letters to other pharmaceutical executives as well, all letters have been ignored.
Obviously, with the announcement of this breakthrough, pharmaceutical SAS programmers were not happy about it ( using SAS, each table typically needs over 400 lines of code (hours of work), using the new syntax the same table can be done in under 20 lines of code (perhaps 20 minutes of work)). So for obvious reasons, pharmaceutical SAS programmers do not want me allowed in the building. (And oh, the pharmaceutical industry continues to pour millions of dollars into one SAS macro library after another, even though CFOs should have known by 1995 that macro libraries do not solve the productivity problem. It was worth trying 25 years ago, but today selling a macro library is like selling a bridge from Brooklyn. CFOs put money into technology projects based upon political reasons, not merit.)


To understand the mistakes that Rice and other pharmaceutical CFOs are making, institutional shareholders and economics columnists in the media need to understand the difference between short-term cost cutting techniques and solving long-term cost-efficiency problems. They are different methods for solving different problems. So long as institutional shareholders and the mainstream media continue to ignore the difference between short-term cost cutting and long-term cost-efficiencies, the cost-efficiency problems in the healthcare industry shall never be solved, because health-care executives are not motivated to solve them.

Short-term cost cutting is easy for the CFO to do. It does not require examination or restructuring of the underlying operations. Typically, it's a layoff announcement. It can also involve shuffling work back and forth between in-house and CROs, or other ways of shuffling costs around. The problem for long-term shareholders (and customers who pay high drug prices) is : short-term cost cutting methods do not raise long-term worker productivity (I'll explain why in a later post). Solving long-term cost-efficiency problems is much harder work, it requires deeper examination of the operations. It is work that pharmaceutical CFOs do not wish to do. When you solve long-term cost-efficiency problems, the benefits for long-term shareholders and/or people buying medicine can be dramatic. But the pharmaceutical industry does not solve long-term cost-efficiency problems, and has not been for decades.

Derica Rice made about five million dollars last year. Other pharmaceutical CFOs made similar multi-million dollar salaries in one year. They do short-term cost cutting, which provides little benefit to long-term shareholders and customers. They do not solve long-term cost-efficiency problems. They also don't talk to most innovators (because most corporate executives are stuck up). Do these people deserve an annual salary of say, $500,000 per year (about a tenth of what many of them make)? They most certainly do not. They should be fired.

My Bias

I'm about to write a series of economics articles on this site, and one of the most important economic guidelines is:

Everybody has an economic(and political) bias (often specific to one's career choice).

In terms of economic incentives, nobody is objective , even if some of us strive to be objective.

So a fair question is: what is your economic bias?

Fair enough, although if you are reading this blog, you already know. But I'll be explicit:

I have the economic bias of an "Angry Innovator".

For decades pharmaceutical shareholders have been consistently ripped off because of the severe worker productivity problems in statistical programming services, problems that could have and should have been solved in the 1990s. I have a solution that brings the cost of statistical table production down by more than 80%. So I'm an innovator.

But I failed to understand just how deeply dysfunctional the politics and economics in the pharmaceutical industry is. It turns out that nobody in the pharmaceutical industry cares about solving worker productivity problems. SAS programmers are not at all happy about this breakthrough (they are protecting their own economic interests). Pharmaceutical CFOs don't answer my letters because they are stuck-up people who think communicating with innovators from the poor or middle class is beneath them.
So I'm angry. So now you know what my economic bias is.

But I have learned some things here, something that theoretical economists and newspaper economics columnists even today do not understand. People will often behave in a superficial or shallow manner (especially corporate executives) and that fact distorts and reshapes the economy in ways that economists don't even try to understand. The health-care industry is particularly bad, and health-care executives simply do not wish to solve long-term cost-efficiency problems and long-term worker productivity problems. And that is a big problem.

Technology Workers are Luddites

I'm going to be putting a number of economics essays on this blog in the coming weeks, because my experience in the American economy has given me unusual insights into the ways in which the American economy is ossified, sclerotic, and gummed up. These attributes of the economy cannot be explained by the mathematical models that academic economists depend upon. And these aspects of the economy are something that I understand much better than corporate executives and newspaper economics commentators. The health-care industry is particularly problematic, and does not get nearly enough attention from the mainstream media. My hope is to change that, because the high cost of healthcare damages a lot of people's lives.

One of the economic guidelines I've discovered over the years is:

TECHNOLOGY WORKERS ARE LUDDITES.

Insulting? Sure. But it is, sad to say, oh so true. Of course, how true it is depends on the context, and depends on when certain economic incentives apply. But it is indeed often the case that technology workers simply do not care about worker productivity, and it is often the case that the economic incentives that drive technology workers are at odds with the economic interest of the people who pay them.

Those of you who know me know that there are more than a few pharmaceutical SAS programmers who are quietly hoping that I get run over by a bus, and that I am not shy in pointing out that their response to a recent computer science breakthrough is the height of irresponsibility. What did I do to earn their contempt? I created a new programming language that runs circles around SAS syntax , in terms of worker productivity, for a certain common type of statistical work. They do not like that, and they do not want me allowed in the building. Because the SAS language has serious worker productivity problems, that costs pharmaceutical shareholders a lot of money. SAS programmers have made it very clear that they do not want this problem to be solved.

So when are computer programmers jerks (or, more specifically, Luddites) and when are they rather nice people? Computer programmers fall into what I'll call cliques (the PHP clique, the C++ clique, etc.), or if you prefer, tribes. Computer programmers are a lot like high school students - they are very cliquish. And their behaviour is all about protecting their clique. If you say nice things about PHP, the PHP clique considers you a friend. If you are a beginner (who doesn't ask questions that are too obviously stupid) who is trying to learn the ropes of a certain clique (say PHP or Javascript) you may find the members of that clique to be very helpful. Welcome to the club!

When are computer programmers downright nasty people? When a new technology platform emerges that can be seen as a threat to their clique (often because of differences in worker productivity), when the new technology shows, by comparison, certain flaws of the clique's favored technology. In that situation, computer programmers have a strong economic incentive to stop change and stop worker productivity growth. In this context, computer programmers are Luddites. In addition, the fact that corporate executives are so ignorant of this dynamic does make the American economy more sclerotic, because corporate executive's ignorance of this behavior means lower worker productivity growth across the economy.

CFOs don't care about worker productivity.

There are a lot of things about the American economy that economists and high-profile journalists (Andrew Ross Sorkin, Jesse Eisinger and so on) do not see. In general, the American economy is far more sclerotic than the economic establishment realizes.

1. Chief Financial Officers don't care about worker productivity.
2. Corporate executives have a lack of respect for the poor and the middle class - this damages total factor productivity.

Regarding worker productivity, CFOs want to do the easy stuff, but avoid the hard slog.
Layoffs are easy: you reduce costs and also reduce the productive potential of the affected departments. Worker productivity remains unchanged. Transferring to an outsourcer is also easy: but the worker productivity problem remains unchanged and you could make things worse. (Executives claim that outsourcers have an incentive to solve worker productivity problems, that is often untrue, think "law firms' billable hours model").
The above is short-term cost cutting, and is of limited benefit to long-term shareholders.

Solving long-term cost efficiency problems and worker productivity problems requires executives to work a lot harder and be more curious. Corporate executives don't want to do that.

Now what about the disrespect for the poor and middle class, if you are a long term shareholder why should you care? Here's why:
A CFO has two choices:
A political insider, almost a member of the corporate elite himself is making a pitch. He has a technology solution of limited benefits (or none at all). The new technology framework certainly does not solve the worker productivity problem. But the fellow has an expensive suit and tie and the CFO is impressed. They shake hands and have a deal.
A political outsider sends a letter to the CFO. He solves the worker productivity problem, in fact his solution increases worker productivity by a factor of five (five hours of work becomes one hour of work). But the CFO is an important guy and he has no time to waste with people of low socioeconomic status. He tosses the letter in the trash.

Are corporate executives really that shallow? Yes they are. I'll give some examples in the next post.

SAS programmers are preventing cost-cutting reforms

If the pharmaceutical industry wants cost-efficiency reforms in statistical programming services, by raising worker productivity, they must evaluate, test, and use alternatives to the SAS programming language.

The pharmaceutical industry has tried to solve this worker productivity problem over the past two decades by writing one SAS MACRO library, after another, after another. When the latest library doesn't solve the productivity problem, write another library. This approach does not work. (How has the cost per statistical table in a Clinical Study Report changed from 1990 to 2012, if at all?)

Senior pharmaceutical SAS programmers do not want any non-SAS product design to be examined, tested, or used. That is their position today, and has been their position for decades. The logical conclusion is that pharmaceutical SAS programmers are preventing cost-efficiency reforms.

The Vilno Table programming language gives much higher worker productivity for complex statistical table production than the SAS programming language does, often a table that needs 400 lines of SAS code needs less than 20 lines of Vilno Table code (and the logic of the Vilno code is also easier to read). That's not a typo: you reduce the amount of code you have to write by 20 times (400 divide by 20 = 20)! Higher worker productivity leads to lower costs.

The evidence, logical arguments, and illustrative examples showing this are rock solid.

The response of most (but not all) senior pharmaceutical SAS programmers is, for all practical purposes:
"We don't care. And we prefer you not talk."

With a few exceptions, the response is apathy, silence, and in a few cases more obvious obstructionism and censorship (moderators of professional internet forums such as PharmaSUG and PhUSE have the power to censor). And they don't have a counterargument that makes logical sense. (But some will say "You can write a macro for that" but not provide the macro library external APIs).

Pharmaceutical shareholders would benefit from such a change. Pharmaceutical SAS programmers do not want change.

Flexibility and Productivity

Vilno Table gives a simultaneous combination of high flexibility and high worker productivity that no SAS MACRO library can come close to achieving.
Senior pharmaceutical SAS developers who have a vested interest in preventing change retort: "A SAS MACRO can solve that problem. Non-SAS product designs are unwelcome in the pharmaceutical industry".

A SAS MACRO library might (even here there is difficulty) work for a small subset of tables that satisfy a very strict set of assumptions. But for the wide variety of statistical tables that statisticians and physicians ask for every day, a SAS MACRO library simply does not work.

The problem is the flexibility of the SAS programming language goes only so deep: whenever you try to solve the productivity problem with a SAS MACRO library, flexibility collapses. Any candidate for a standard SAS MACRO library is rigid and difficult to use - it lacks flexibility.

The Vilno Table programming language has high worker productivity: the worker can do a complex statistical table in 20 lines of code (instead of spending 3 hours writing hundreds of lines of code, as with SAS). But Vilno Table combines high worker productivity with high flexibility: summary statistics and advanced statistics on the same page, advanced statistics from different models in the same table, different parts of the table are allowed to use different data sources and different computations, and so on.

Take the framework of tree data structures, blend it in a very careful manner with the types of concepts in a statistician's head just before she says "I want you to code for me a table that's BLA-BLA-BLA", and what you get is a product design that completely blows the SAS programming language out of the water. In terms of worker productivity, it's not even a close call, even if many senior SAS programmers do not want to admit it.

Chi-Square and Linear Model Connectors

The current software implementation of Vilno Table has just two syntax connectors (at the moment): the chi-square connector and the linear model connector. Recall that summary statistics (N, %, median, etc.) do not need such connectors. (The extern-syntax connector(for rare procedures) is not yet operational.)

Consider briefly the implications of the chi-square connector and the linear model connector. The chi-square procedure is (just barely) more advanced than a summary statistic. But the structure of the output data is very simple. So the small code module that is the chi-square connector is simple and small. Other procedures that are just barely more complex than a summary statistic (Fisher's exact, t-test, etc.) will only require a connector module that is quite similar to the chi-square connector.

The linear model connector is quite different - it's a proof of concept (like the New York song: if you can make it here, you can make it anywhere). The structure of the output data from a linear model procedure is nuanced. This means that if I can write a fully functional and correct syntax connector for the linear model (which I have already done in the current software version), then it is very plausible to do so for any statistical procedure: mixed model, factor analysis, survival analysis, etc.

SYNTAX CONNECTORS

With as little as THREE lines of Vilno Table code, the end user can draft a statistical table with a mixture of summary statistics and advanced statistics (p-values, etc.), a table that in SAS needs hundreds of lines of code and hours of work. How can this be possible? Part of the answer: syntax connectors.

A syntax connector is a small code module attached to the main conceptual compiler. Each advanced statistical procedure (chi-square, survival analysis, linear model, etc.) needs a syntax connector. When the syntax connector for a statistical procedure (say, linear model) is available, then you can request the output statistics directly in the column and row statements ("fvalue" is F-statistic, "pval_pw" is p-value for a pairwise comparison, and so on), which makes your work so much easier. (For summary statistics, syntax connectors are not needed).

[ To some degree, this blog post is inside baseball: the end user drafting tables won't read or write syntax connectors - they are part of the Vilno Table product. But if you are asking how this can be possible, this blog post is for you. ]

The current version of the software has a chi-square connector and a linear model connector under the hood. What that means is that the conceptual compiler "understands" how to handle chi-square statistics and linear model statistics (including F-statistic, regression coefficients, least-square means, pairwise difference estimates). Of course, more syntax connectors shall be added (survival analysis, mixed model, and so on). In addition, an all-purpose extern-syntax connector shall be added for very rarely used statistical procedures.

**************************************************

The extern-syntax connector will allow for statistical tables that include advanced statistics from statistical procedures that do not yet have a syntax connector written for them. This will be especially useful for statistical procedures that are highly specialized (and not widely known) but have an implementation in one of the computational back engines that Vilno Table can be connected to (R, S-Plus, SAS/STAT, etc.).

For the most part, using the extern-syntax connector will involve writing a paragraph of code in the native syntax of the statistical procedure computational back engine. In addition, appended to the end of this paragraph is a line of code for meta-data description (telling the conceptual compiler how to expect the output data to be structured). [ By contrast, for statistics that have regular syntax connectors installed, the end-user does not have to write or read any native syntax.]

Computational Back Engines

The software implementation of Vilno Table is (mostly) a front end, it needs to be connected to two computational back engines: one engine for data transformation, data preparation, and summary statistics and the other engine for advanced statistical procedures (linear model, survival analysis, etc.).

The front end, a "conceptual compiler", controls the two computational back engines: giving them lower level code files (that the worker does not have to look at) to execute and gathering the results to put in a statistical table.

The choices for the data transformation back engine include:
1. Vilno Data Transformation (VDT). A low-cost open source replacement for the SAS datastep (as well as proc transpose, proc means, and proc sql (for many-to-many joins))
2. SAS/BASE (SAS datastep, proc sql, proc transpose, proc means)
3. SPSS (possibly), depending on demand
4. others , depending on demand

The choices for the statistical procedure back engine include
1. S programming language implementation (R or S-Plus)
2. SAS/STAT (procedures for linear model, survival analysis, etc.)
3. SPSS, depending on demand
4. others, depending on demand


At this time, the current implementation of Vilno Table uses low-cost choices for the computational back engines: VDT and R. (Remember: the worker does not have to write or read any VDT code or R code to produce statistical tables, Vilno Table does that work under the hood.)

Also, when a later version of Vilno Table is configured to two or three statistical procedure back engines (at the same time, on the same hard drive), which is very doable, then you can easily do a statistical table with p-values and other statistics from R and S-Plus and SAS/STAT, all on the same page.

changing gears

SWITCHING GEARS, WHEN IT'S NEEDED

The history of programming languages show that when the syntax of a programming language is at a higher level of abstraction, you can get a huge increase in worker productivity, but situations where a lower level of abstraction is still needed can occasionally occur.

Programming languages such as SQL, SPSS, SAS, VDT (Vilno Data Transformation), and Vilno Table are data paragraph scripting languages - consisting of paragraphs of code that read and create datasets. R is often used as a data paragraph language (Python can be, but it's purpose is more general).

The worker can use source code files that consist of multiple paragraphs of code. Different paragraphs can use different levels of abstraction and different syntaxes, as long as the datasets created have a common format (i.e. the row-and-column format used with SQL). For Vilno Table, the end user can write paragraphs of Vilno Table code (such as the table paragraph), and the end user can also write paragraphs of code in the native syntax of the computational back engines, if needed.

Fine. But WHY would the end user want to do that? Because statistical programming is a multi-stage work process: data analysis, data reporting, and multiple stages of data transformation (data preparation). In particular you have:

1: Early-stage Data Transformation
2: Mid-stage Data Transformation
3: Table-stage Data Transformation
4: Statistical Table Production
5: Rarely used niche statistical procedures

Table-stage Data Transformation: Most of the data transformation code you see in statistical table SAS programs is table-stage data transformation, data transformation that's an integral part of statistical table formation. You do not want workers writing table-stage data transformation code by hand - it's a big waste of time and money. The table paragraph handles such data transformation "under the hood".

Statistical Table Production: Use the table paragraph, and the statistical table needs less than 15 lines of code. Use the SAS programming language, and the same table needs 400 lines of code.

Niche statistical procedures: For very common statistical procedures, Vilno Table has now or will have syntax connectors installed - you can include advanced statistics directly in the table with just a couple of lines of code. But for rarely used statistical procedures (that lack a syntax connector), it will be necessary to write a paragraph of code to call that statistical procedure in the native syntax of the chosen statistical procedure back engine (typically, R, S-Plus, SAS/STAT).

Data Transformation (data preparation, data cleaning):
You can always use the old-fashioned syntax of the data transformation computational back engine to do data transformation, data preparation, and data cleaning: usually either VDT or the SAS datastep (and someone who knows one can learn the other very quickly).
Data transformation needs are diverse and hard to categorize, but I divide it into early-stage data transformation and mid-stage data transformation.

Early-stage data transformation is often called "data cleaning". Typically, you tell the statistician what's wrong with the data (partially missing data, unusual visits, etc.) and she decides what data adjustments to make prior to analysis. This is tedious work and a product design that multiplies worker productivity for data cleaning is still an unsolved computer science problem. It's unusual for data cleaning code to be in statistical table programs, it's supposed to be finalized in the programs that create the ready-for-analysis datasets from the raw datasets. For data cleaning, there is simply no choice: use old-fashioned syntax (VDT or SAS datastep), with low worker productivity. [This is a big branch of applied computer science in which I am active, but in which academic computer scientists (and employees of SAS Institute ) have accomplished absolutely nothing over the past 20 years - that's worth noting. There is only one serious researcher in this part of computer science, and that is me.]

Mid-stage data transformation is more geometric, almost like a generalized transposing. Calculation of % change from baseline is an example of mid-stage data transformation. Usually mid-stage data transformation is done at the end of programs creating ready-for-analysis datasets, or at the beginning of a statistical table program. A worker can often do mid-stage data transformation (with higher worker productivity) using the parity paragraph, using old fashioned syntax (VDT or SAS) is another option.

My blogs: what goes where

This is my fivetimesfaster blog, for specific issues of the Vilno Table programming language.

For more general issues of applied computer science, I have another blog at

http://comp-sci-stuff.blogspot.com

There I discuss more general issues of statistical software, data transformation and data cleaning.

Vilno Table Programming Language White Paper

Here is a link to the
"Vilno Table Programming Language White Paper"

It is a .PDF file stored in a Google Documents area :

https://docs.google.com/open?id=0B2oCqHJ9cxhCNkp2T0dnMFFBdlE

Rather than viewing it in the Google Documents area, download it and open it after downloaded, the viewing within Google Docs of a .PDF file might show a couple of pages as blurry.

The first 8 table examples in the appendix are the most intuitive way to get a feel for how this language works.

Vilno Table, Real Examples of How It Works

These are actual examples actually produced by the software product (not yet version 1.0, but very functional, as you can see, the technology works).

Because the tables produced by the software are not so easy to put into a blog post, I am using a link to a document in "Google Docs":

https://docs.google.com/document/d/1O21VX4MCwcs2u_1eCwx5An-Di9NDld0-GCp0yHUYZLA/edit

So click on it, and you will see the examples from the appendix of the white paper.

Each example is 2 pages: the code (usually less than 15 lines), and the table that it produces.

I am having difficult choosing landscape/portrait for each page, so every page, unfortunately, is landscape. When you upload to Google Docs from a word document on the hard drive, it throws some stuff out, like page-by-page choices for landscape/portrait

More Examples of Vilno Table

Here I show a few examples of rather simple statistical tables. Vilno Table has an enormous worker productivity advantage over SAS for complex, customized, picky statistical table requests. But it's also easy to use for simple tables. The more complex and picky the statistical table request is, the greater the advantage that Vilno Table has over SAS (and Excel) in terms of worker productivity. (Therefore, for very simple requests, the difference in productivity is less).

Most of these examples use only summary statistics (the last example will be a one-way ANOVA). Each example has only one available dataset and essentially one analysis (Vilno Table can produce a table using multiple data sources and different analyses, but these examples are simple).

This is code that describes the available datasets. Here, there is only one dataset, the PATINFO dataset :

inputdset asc a/PATINFO site patid trtgrp gender race happy weight age ;

( trtgrp means "Treatment Group", happy is a categorical outcome variable).

This is just a two-way frequency table:

denom trtgrp ;

col trtgrp*( N % ) ;

row happy ;

This is the same thing, but add a chi-square p-value in the upper right corner:

( I add a model statement, and add the word "pvalue" to the column statement )

denom trtgrp ;

model chisq(trtgrp*happy) ;

col trtgrp*( N % ) pvalue ;

row happy ;

This is assorted summary statistics , for certain demographic subgroups:

col all gender*race [age<65] ;

row N mean(weight) std(weight) median(weight) ;

Okey-dokey, let's try one slightly more advanced example, a one-way ANOVA (well, technically "age" is a continuous covariate). Lm stands for "Linear model".

model lm( weight ~ trtgrp -1 + age ) ;

col trtgrp*est all*( est_pw("60mg"-"Placebo") pval_pw("60mg"-"Placebo") ) ;

row all ;

What you get is least-square mean for every treatment group, and just one pairwise comparison.

Let's make the row and column headers easier to read with:

label trtgrp "Treatment Group" est "Least Square Mean"

all "60mg vs Placebo" ""

est_pw "Difference" pval_pw "P-value" ;

To the dataset description code at the top, for the linear model, I'll need to add:

categorical trtgrp ;

continuous weight age ; (this extra description code was not needed for the summary statistics tables)

The above 4 examples are a lot simpler than most of the examples in the appendix of the Vilno Table Programming Language. The above examples do not show the full flexibility of Vilno. They show it's easy to produce tables that should be easy.

Just one more example, before I go, the next table is several frequency cross-tabulations, each one with a chi-square p-value on the right side, stacked vertically (each row section has a different row category, but the column category is always treatment group.

Several categorical tests, with N and % , and the p-value in the right column, like a baseline characteristic table. Again, most of the important stuff is in the model statement, the column statement, and the row statement.



model chisq(thisrowcat*trtgroup*N) ;


col (trtgroup all)*(N %) pvalue ;


row gender race age_group ;



This is fairly similar to table A1 in the Vilno Table white paper. When the current beta version has cr-modifier statements added to the parser, a later version will be able to put continuous and categorical statistics into the same column, but not yet.
(A baseline characteristic table crams a lot of stuff into the same page, so N and % must share the same column with mean and std. deviation).

AE table in 11 lines of code: Why it is possible.

Here I explain the 11 lines of code that produce the AE table in more detail, to clear up some misunderstandings:

Here is code for table A3 in the Vilno Table Programming Language white paper, an AE table with the typical bells and whistles:

title "Table A3: AE table, with chi-square (75-patient dbase)" ;

directoryref a="/home/robert/test" ;

inputdset asc a/patinfo3 patid trt 1*(patid) ;

inputdset asc a/advevt3a patid bodysys prefterm ;

thing pat uniqval(patid) a/patinfo3 ;

n~n(pat) ;

printto "/home/robert/test/outp01" ;

denom trt ;

model chisq(thisrow?*trt*n) ;

col (trt all)*(n %) pvalue ;

row all have(a/advevt3a) bodysys*(all nothave prefterm) ;

(Please note: patinfo3 and advevt3a are names of datasets, the patient_info dataset and the adverse events dataset.)

(Please note: trt is the variable name for treatment group (here 3 groups), bodysys and prefterm are, obviously variable names for body system and preferred term.)

As I've already said, the best way to explain and learn this language is to focus attention on 3 lines of code: the model statement, the column statement, and the row statement (which are the last 3 lines of code in the above example). It is by reading these 3 lines of code that you can see what the statistician is asking for. Most of the analysis logic is in these 3 lines. If you do many statistical tables off of the same database, the other lines of code require little modification.

But to dispel some confusion, I will go through all 11 lines. Naturally, I begin with the model, column, and row statements:

Look at the column statement:

col (trt all)*(n %) pvalue ;

You have an N and % column for each treatment group (there are 3 treatment groups in table A3). You have an N and % column for all patients together. In the right-most column, you have a p-value from a simple categorical test.

Look at the row statement :

row all have(a/ae_dataset) bodysys*(all nothave prefterm) ;

Going through each piece of the row statement in order, what it does:

row

all -> row with grand total N for each group

have(a/ae_dataset) ->

row with number of patients with any AE at all (no matter what body system or preferred term)

bodysys*( -> for each body system, a set of rows that include the following:

all -> row for number of patients with AE for this body system

nothave -> row for number of patients who do NOT have an AE (for this body system)

prefterm -> for each preferred term, number of patients with AE

This AE table is a famous example. It is not a beginner's example for two reasons: for each body system there is a row for number and % of patients who do NOT have an event ; and the request for a categorical p-value for EVERY row in the table (for every preferred term). For these two advanced issues, two advanced keywords in the programming language are used: NOTHAVE and THISROW? respectively.

In a very simple example, the model statement would look like this:

model chisq(gender*race*N) ;

The column and row statements, which are at the center of this programming language, use a "visual tree" method: you write a syntax that becomes a tree data structure, this tree data structure becomes a visual display at the top of (and left side of) the statistical table. The terms in the column and row statement are called TABLE FACTORS, which become nodes in the visual tree.

THE TREE DATA STRUCTURE IS A COMBINATION OF DETAILS THAT SHOW WHAT THE STATISTICIAN IS ASKING FOR.

With elementary tables, the table factors are: categorical variable names, boolean expressions (such as [age<65]), names of statistics( such as N % mean std(std.deviation) pvalue est(estimate for least square mean) fvalue(F-statistic) and so on), plus the ALL keyword.

With more advanced tables, more advanced keywords in the programming language might be used: HAVE NOTHAVE THISROW? THISROWCAT .

Lines 1,2, and 7 are simple, obvious, and nothing new.

Lines 3,4,5,6 are input dataset description code. That includes the thing statement, which is a new innovative feature: you have to define WHAT you are counting, and later specify WHERE you are counting. If you want to count people AND count event records (using ae_record_idnum if in the dataset), you could have TWO thing statements for the same table (I've seen it asked for(tracking before database lock), but it's not that common a request).

The keyword "inputdset" means: here is an input dataset which I describe here (obviously there is a PATINFO dataset and an AE dataset, as you expect). I'll get into data source identification later, but if you have two datasets, it's pretty obvious to the compiler.

The denominator statement is "denom trt ;" . Is that fairly obvious? It actually is part of the table paragraph, with the last three lines.

This is not a macro library, it is a new language, and it can produce a huge variety of different tables. (I still haven't shown you the linear model example, table A2, that is going to blow your mind!).

Robert Wilkins

Produce Statistical Tables Really Fast

So what is this new product?

It's an implementation of a new statistical programming language, now up and running in beta mode, that produces statistical tables really fast , with a really small effort in coding.

Since the software is in beta mode, I've already run some examples:

Typical AE table (N % p-value(chisquare) , each body-system, patients who do not have (each bodysys), and of course body-system*preferred-term) ->

That's 12 lines of code, baby! (SAS, you need around 600!)

Typical baseline characteristic ( but no continuous ), 3 categorical variables going down, one category across(treatment group) , N % p-value(chisq) ->

That's 9 lines of code , man! (SAS, hard pressed to get it under 400)

Linear model example: pairwise (t-statistic and p-value, ends up being 4x4, in the leftmost section of paper), model coefficients( t-statistic and p-value, one column 4x1 ), and F-statistic (with p-value, in the rightmost part of page) ->

That's 11 lines of code , man (SAS, hard to say, probably 400 lines of code, or more)

This is not a macro library. This makes use of tree data structures in a much more sophisticated way than SPSS/SAS/SQL.

The current version is on Linux and outsources(gives work to do) to vilno data transformation and R. This DOES NOT mean that the product forces the end user to look at vilno code or R code. A version that outsources to SAS/BASE and SAS/STAT is not yet in beta mode

Five times faster than what???

Develop complex statistical tables, upon request, five times faster than the SAS programming language.

Also, develop these tables much faster, with less chance for error , than with a spreadsheet, with a better record of what you did if you come back to it three months later.

Also, much more expressive and flexible for sudden requests and changing requirements than any language macro system can be.

That's what five times faster means.