Why did Indian scientists and mathematicians fall behind China?

The short answer is they didn't really.

What happened was China decided Science & Maths were good things at about the same time that India decided that Stupidity & endless Moralising were even better things provided someone else kept refilling one's begging bowl.

Still it seems strange that India had a head start and chose, quite democratically, to throw it away.

The fact is, some Indians started studying Western Mathematics and Science about fifty years earlier than any Chinese.
By 1850, there were some Indians teaching in private Colleges, or Institutions funded by the East India Company, who sought to publish and circulate their findings amongst Western scholars.

Master Ramachandra, of Delhi College was the most prominent; his name has come down to us as 'De Morgan's Ramanuja'.

Speaking generally, English speaking Indian savants of the period faced acute social and financial pressures. On the one hand, they could secure a safe bureaucratic career by doing 'applied' work rather than 'pure' theory. However, if they were really good at their job, then, like Radhanath Sikdar, they would be denied transfer to the more lucrative branches of officialdom.

On the other hand, if they wished to gain employment as a College instructor, they had to produce credentials of a costly and inutile sort before sacrificing their working lives to the intolerable drudgery of cramming the mandated curriculum into heads destined to be sacrificed to a like bureaucratic maw. Ramachandra may have hoped that conversion to Christianity, or loyalty to the Company would help him. It didn't. It marked him as a fool. Mastery of Sanskrit mimamsa could raise a man up in the estimation of John Company and make his fortune for him in the law courts. Mathematics held little prestige. It is noteworthy that Ramachandra made no further contribution to Mathematical research after his conversion. I suppose it was no small thing to have survived the Mutiny, when converts like himself were butchered. Still, it is sad to think of him sinking back into the drudgery of the scholastic round- he, who had once thought to overturn the tyranny of the Mullah and the Padre! was now as but their puerile drill-master.

De Morgan, in his preface to Ramachandra's book, seeks ineffectually to differentiate Hindus (he calls Ramachandra a Hindu because he uses the term in a racial sense) from naked cannibals- albeit, not the South Indian Hindus, amongst whom De Morgan himself was born, who struck him as being identical in appearance and culture to such incorrigible savages as, I suppose, were constituted by those Tasmanians whom European Enlightenment was busy exterminating about that time

De Morgan's point is that Mathematics can't progress save by speculative means and the methods of pure a priori ratiocination. Hindu stagnation in mathematics illustrates his thesis. Their racial inability to maintain the spirit of pure inquiry is what led to the ossification of their Mathematics into a set of 'practical rules'.

Clearly, the methods of pure Mathematics or Theoretical Science can only be inculcated in minds that work on their own. But the same is true of linguistic ability. It is not enough to be able to convey your own wishes in a particular language or to ensure that they are complied with by a fluent speaker. Unless you can wish to say anything anyone else might convey, you haven't really mastered it at all.
Why? Intention is a register in intensional discourse and can only be discriminated by an equal intention.

De Morgan's thinks that some external entity can generate a particular type of cumulative creative thought in a Society, by initially offering some impetus or encouragement and then afterwards rewarding merit. This has a Darwinian ring. If we want to breed a race of monkeys that can type Parliamentary speeches, we initially settle a group of monkeys in the Visitors' Gallery at Westminster and to encourage them to press keys on typewriters in return for bananas. We periodically check on the monkeys to see if they are taking down Mr. Gladstone's speech correctly and suitably reward those monkeys which come closest. While this is a theory which certainly explains much in Hansard that must otherwise remain opaque to the eyes of even the most Gadamerian hermeneut,  I confess, it isn't really convincing.

Why?

 It violates De Morgan's own principle of induction.  A series with a particular property can't be defined inductively if there is an exogenous determinant of fitness for inclusion. Even if the monkeys end up doing a better job than any human, so long as monkeys don't talk or think like Mr. Gladstone they can't be said to have transcribed his speech. There is some causal connection between what they are doing and what Mr. Gladstone is doing. But that connection is not the same sort of connection as subsists between a human transcriber and a human speaker. For a start, no monkey can bear witness as to the accuracy of his transcription. Thus, if Mr. Disraeli calls upon Hansard to show that Gladstone, on such and such a date, declared his willingness to eat his own cat, contingent upon an eventuality which has since come to pass, no monkey can confirm that this is exactly what happened. Thus, if Gladstone gets out of his commitment by eating his head gear rather his own pussy, no charge of bad faith can be sustained against him.

More generally, there can't be a 'Chinese Room' type examiner in a Turing Test of either mathematical method or linguistic capacity.

Why?

Well, either the monitoring system is itself an equal or superior application of the method that is being tested, the capacity that is being assayed, or else it can be gamed. (There is an argument from Razborov & Rudich, re. 'Natural proof', which explains why.)

If monitoring capacity is equal or superior and interacts with what it examines, it isn't exogenous at all.

You can get a 'zero knowledge proof' for information but not one for intensional consistency.

There can't be a 'hands off' approach to Paideia.

Victorian England knew very well that this was the weakness of the 'capitation' system.

If you pay Schools to cram a certain curriculum, all you get is better cramming and lower comprehension. 

The same problem arises at the other end of the social spectrum.

If you discriminate mathematical talent through demanding Examinations- e.g. those for the Cambridge Tripos in which top scorers received the title of 'Wrangler'- you don't necessarily get creative mathematicians. Indeed, the system may be adversely selective in this respect. The young man who has gained a sinecure by holding his nose to the grindstone may choose to stick it in no theoretical nook where it fails to earn an equal or immediate reward. The fact that the English Tripos was more 'mixed'- i.e. physicalist and computational- than the new 'purer' methods of analysis gaining ground in France and Germany- meant that the great British scientists of the second half or the century, most of whom had done the Tripos, appeared to confirm, not challenge, a common sense view of the world as pervaded by some electromagnetic ether or other guarantor of sound Newtonian substantivism.

De Morgan, writing at the mid Victorian noon-tide, could not envisage how swiftly his own country, and its Capital City where he himself taught, would be eclipsed and rendered provincial with respect to his own profession. This passage of his holds a melancholy interest-

Why did London sink while Berlin rose?

The answer is, firstly, that German Universities competed with each other for able Professors and Students. Endogenous competition promotes 'run away' effects. Savants are more ready to go where pure speculation takes them because they answer to no exogenous monitor or system of reward. The Germans increasingly adopted the 'Seminar' system- popularised by Indological philology- which in turn meant future competition would be Research based thus reducing the importance of exams or disciplinary regulations. In part this was because students had more power because they could more easily 'vote with their feet'. However, it was also the feeling that Germany was obliged to find its Empire in the Mind- not the Indies- in order to equal or surpass the glory of England or France or Russia, which drove the startling ascent of German academia.

In Britain, many pedagogues, having exhausted their intellectual promise in the slog towards a Fellowship, idled away their time in petty intrigue while the Port went round. Meanwhile, the profits of Industry and the tributes of Empire piled up so quickly against the wolf at the door that none questioned a habitus of decreasing effort and increasing rewards. Though the day was short and of frost and fire, they slept before evening.

In Germany, by contrast, there was a nexus between Universities and the newly emerging knowledge based industries whose fate was tightly interwoven with the Financial institutions which supported their growth. This nexus relied upon and was relied upon by the State. However, all too palpably, that State was so menaced or menacing that nothing was really secure. Speculative thought carried less risk and greater reward precisely because it allowed one a sort of 'internal emigration' to, not a changeless Platonic realm, but something dynamic and Heraclitean.

Writing in the 1850's, before Germany had begun its Industrial ascent, De Morgan may be forgiven  a certain complacency. Yet, by that iron law of irony every Zeitgeist imposes upon its own arbiters- more particularly Professors of Logic- he was bound to say the most foolish thing possible about his own country while writing about a native of one of its far dependencies. 

Thus, he paints a rosy picture of the apple cheeked English yokel, or soot begrimed Scottish mechanic, turning joyfully to the pages of 'Tensor Calculus Tidbits' as a way of recruiting his spirits before turning back to his wearisome task. No doubt, women and children employed in the textile mills could no longer 'bend over the Principia of Newton' as was the wont of English weavers in days of yore, but surely the Bobby on the beat, or the postman delivering letters, or the rag & bone man on his rounds, paused every so often at the Newsagents to keep abreast of the latest developments in Galois Theory or Algebraic Topology.

Turning aside from De Morgan and returning to India, we can but sympathise with the Sikdars and Ramachandras whose youthful promise was quenched by the drudgery of bureaucratic or pedagogic employment. In Bengal, a greater concentration of wealth and the emergence of a highly emulous bildungsburgertum did create a better atmosphere. Calcutta University was intended to be a first rate institution with high entrance requirements. It pushed up standards but also contributed to a sense of failure and inadequacy in the rising generation unable to cram such indigestible fare for the meagre reward of a Babu's berth.

Thus, though on the one hand, by the turn of the century, India was getting its share of Senior Wranglers, there was also, on the other, a disproportionate and larger recoil from, and rejection of, Western paideia. 

In any case, it was lawyers- not professors or bureaucrats- who were making the big bucks. The Mathematical Sciences were at best a hedge against a decline in Social Status. They were certainly not a route to the top nor did they hold the key to a Political Revolution which might have the equivalent effect of saving one from poverty and powerlessness. On the contrary, once even London returned barristers gave up their horsehair wigs for khaddar caps and exchanged Macaulay's orotund English for senile or sententious versions of the Vernacular, it was clear that the pure Sciences would play a diminishing role in independent India. Thus the Bose for whom the boson is named made no further contribution, not because of an obstruction placed by European racism, but because of the distraction posed by local politics.

Were there any creative Indian mathematicians prior to the advent of the great Ramanujan? 

Apparently, only one- Syamadas Mukhopadhyaya- but the general level of instruction in Mathematics- and later Statistics- continued to improve- which is why Ramanujan could receive a leg up from Indian amateur mathematicians.

Indeed, even in 1950, Indian students in Statistics and O.R were ahead, not behind, their coevals in America. Still, the fact is, Maths declined in prestige. It actually became more difficult for a talented mathematician without a degree (which would have involved cramming a lot of Milton and Botany and other such useless subjects) to get a University appointment. Influential patrons like Professor Hardy could help but that sort of patronage could also be counterproductive. Andre Weil says Hardy spoiled many Indian mathematicians. They continued working on problems which had ceased to be of interest. But Weil himself was not to last long in India. Genius or not, he was after all an unclean White man and unclean White men were ceasing to matter. Being the right caste or belonging to the right clique or having married into the right politically connected clan was what mattered. Even if you got your Professorship, you'd be pensioned off at 55- as happened to Ananda Rau.

The case of D.D Kosambi, who worked with Weil at Aligarh, is instructive. The poor fellow, though a Harvard man and well connected, could not hack academic politics and, like his father, ended up taking a lowly lecturer's position in his native place. His boss was a Cambridge Wrangler who, true to form, never touched Mathematics again.  Kosambi could scarcely flourish having been so cut off from his peers. He was sacked because his students couldn't understand him. They just wanted to pass their exams and get jobs. How or what was he meant to make them understand?

At this juncture, Knowledge based private industry came to his rescue. He was offered a high salary to do pure research for the Tata Institute. I believe he was the first Indian to be sent abroad for work to do with the development of the electronic computer. Nehru, who was socially at ease with the scions of wealthy industrialists who had received a similar education as himself and whose families had financed Gandhi, created some Research Institutes funded by the Government but run by these plutocrats. The results were not particularly impressive. However, it must be said, purely Government sponsored Institutes seem to have performed even worse.

Perhaps, if there had been wholly private investment in Fundamental Research, the story would have been different. In the case of early work on the electronic computer, it took many years to pay off. But pay off it did and on a scale few could imagine.

 For Kosambi, however, there was the more immediate attraction of Chinese Communism and posing as some sort of Marxist historian cum philologist.

Around this time, he published a proof of the Reimann hypothesis in some Indian Journal devoted to Agricultural Statistics. Was it all a practical joke? Perhaps. Still, the damage was done. Kosambi is remembered as some sort of jhollawallah retard- more particularly when he published a second proof of the Reimann conjecture a couple of years before his death.  Why did he do so? The Communist J.B.S Haldane- who took Indian Citizenship for some silly political reason- was also behaving oddly around this time. The truth is, posh people who think they are smart go stir crazy in 'Soft State' Bureaucratic milieus. Ultimately, I suppose, it is the hypocrisy that gets to them. In proper Communist countries, you have brazen lies and people getting shot. That can be quite invigorating for Leftists. In India nobody got shot and nobody bothered to lie. Everybody just talked nonsense incessantly.

The pity of it is Kosambi could have been a great mathematician and an effective populariser of Science. Interestingly, an early contribution of his has secured him joint credit along with Cartan & the (slightly later work of the ) Chinese mathematician, Chern, for what is increasingly called KCC theory which, for a reason De Morgan pointed out, bears some resemblance to Master Ramachandra's work. But, thereafter, Kosambi went increasingly off the rails.

As he said to his friend, Norbert Weiner, who greeted him as a 'Wise Man from the East!'  he was just a 'wiseguy from Cambridge'.

Why? India was falling behind in not just Science and Mathematics but also the ability to write sensibly about History or even poetry- even verse as pellucid as Bhratrhari's. Everything had just come down to cramming and vomiting so as to get a bureaucratic berth. For alumni of either Cambridge, something more was required- viz. hypocrisy and fraud. Statistical methods can't prove the Reimann conjecture. It is hypocritical and fraudulent to pretend otherwise, but who the hell cares? Let's just go ahead and do it anyway. That way we will be trusted as ideological allies rather than 'useful idiots' from whose eyes the blinkers might at any moment drop.

It may be objected that Indian savants could have developed a professional ethos and subjected each other to the discipline of peer-review.  But, their economic base was so slender. Indian mathematicians and Statisticians did club together and set up periodicals but how could they assure quality control? Who will tell the V.C that his paper on Gandhian Mathematics- based on going on hunger strike till intractable equations experience a change of heart and solve themselves- is worthless drivel?

In any case, Indian politics had been on a increasingly irrational trajectory from about 1917 onward. People who spoke the same language and who lived in the same village suddenly declared that they actually spoke completely different languages and belonged to wholly different nations. Ideology could not provide social cohesion. The only thing two ideologues of whatever stripe could agree on was that everybody else was an evil schismatic bought and paid for by some sinister foreign power. Stalin came to loathe the Indian Communists because they kept demanding he kill every other Indian Communist- more particularly those with whom they themselves enjoyed the closest or most cordial relations. Why couldn't they just knife each other? It was all just a case of Brahminical disdain for manual labour. 

The Soviets did take a lot of Indian students and treated them generously. However, these students were convinced that the White man would not pass on his secrets and wanted to keep the brown man permanently ignorant. So, Indian students astonished their Soviet hosts by the extent of their corruption and concupiscence. By contrast, Indians who paid their own way to the States worked their asses off and fast traced to citizenship.

One reason Universities and Govt. funded Research Institutions had to start treating Math mavens properly was that they now had options. If you harass them too much, they'll just book their ticket to America and never look back.

Still, it is the growth of indigenous private sector Knowledge based industries which are dependent on a newly assertive and self respecting professional class of academics and researchers which will restore Mathematics to its rightful place in India. 

What about China? They started later than us but, because they invested heavily in getting the best instruction, they started producing great mathematicians very quickly. Furthermore, they were not obsessed with paper qualifications. 

Take Hua Luogeng. He had no formal qualifications. He happily turned from pure to applied research when the needs of the Nation- or Chairman Mao's whims- demanded it. Interestingly, it was Norbert Weiner- a colleague of D.D Kosambi's father- who enabled Hua to study with Prof. Hardy in the Thirties. Hua survived the Cultural Revolution because Mao perceived him as an 'applied' man of the people, not a bourgeois purist.

One of Hua's friends was the utterly brilliant Shiing-Shen Chern who settled in America. However, after Nixon's trip to Beijing, people like him were able to reconnect with the Chinese mathematical community which had suffered so much during the Cultural Revolution. Hua played a role in making this possible. Once America and China had restored their traditional friendship, the diaspora helped lift up the native country. This was a symbiotic, not parasitic relationship. As China became wealthier, a two way relationship has been established. The next Shing-Tung Yau may well be of diasporic origin but live and work in China.

Of course, it should be mentioned, China- unlike India- had made a point of importing the best Professors on high salaries as soon as it became a Republic. Universities were truly National institutions- not a method of rationing jobs in the Bureaucracy. Furthermore, unlike Indian Research Institutes where the bureaucrat or socially prominent insider held the power and the technical staff were expected to display sycophancy not results, China followed the Soviet model of protecting Defence related Technicians from the Ideological Commissars and, in some cases, even shielding them from the excesses of the Cultural Revolution.

It is ironic that the 'Mandarin' mentality prevailed in India as 'Competition Wallahs' lorded it over Scientists while the country which devised the system of Competitive Exams developed a dynamic indigenous Defence Research capacity. It must be said, that not all Indian Research Institutes are moribund. Clearly, it has some remarkable people and very successful programs. However, this happy outcome was the product of a lot of skillful behind-the-scenes politicking. This was not the initial expectation in the Fifties. Many Scientists believed that Nehru would set up a Technical Cadre of equal rank to the Administrative Cadre. The manner in which this initiative was strangled by the IAS makes for horrific reading. Fortunately, military and economic setbacks in the Sixties weakened the grip of the 'Babus' so some useful work got done. Strangely, it was the Hindu Nationalist party- whom we all firmly believed to rustic retards- which carried forward the 'Nehruvian' vision. The Leftists were no use at all.

China and India have changed a great deal over the last thirty years. Both now have a cohesive indigenous class of professional scientists and mathematicians who can be safely left alone to get on with their job. Increasingly, they will be self-funding. India can quickly acquire World Class institutions, of a type China already has, simply by disintermediating the U.G.C and giving the private sector an increasing role. Government money can't come without strings attached otherwise bogus or corrupt institutions- like Nalanda International University under Amartya Sen- will turn into scandalously mismanaged money pits.