By the beginning of 1903, Matisse’s fortunes had hit rock bottom. His father-in-law was publicly disgraced and imprisoned for a crime he did not commit. His wife had taken to her bed, and his family was destitute. Matisse had been forced to give up the strange, incomprehensible, frankly unsalable work that interested him, and to try to earn a living for his extended family by switching to more conventional flower paintings and portraits.

At this point, by his own account, he fell in love. In the 15 years it has taken me to write his biography, I have learned to pay attention whenever Matisse used the language of love. Passion for him meant painting. In 1903 he lost his heart not to a woman but to a length of fabric.

Over the next six years, Matisse conducted successive affairs with little bits of stuff. Each in turn triggered a series of experimental, sometimes explosive pictures. The first of these affairs, in 1903, was with a piece of cotton material spotted in a junk-shop window from the top of a bus in a sleazy part of Paris on the Left Bank of the Seine. Matisse fell for it at first sight.

That square of tattered, blue and white, flower-patterned cotton is the key to the exhibition that opens next week at the Metropolitan Museum: “Matisse, His Art and His Textiles.” Matisse owned it for more than 50 years. He called it his “toile de Jouy” and took it with him wherever he went, hanging it in every studio he ever had and painting it as obsessively as any human model. It kept him company until he died in 1954.

The idea for this exhibition first came to me from looking at the extraordinary paintings Matisse made of his “toile de Jouy” during the first decade of the 20th century. From the great “Harmony in Red” (which scandalized the Paris art world) to the various versions of “Still Life With Blue Tablecloth,” these were revolutionary canvases, meant to blow apart the naturalistic tradition of Western painting and change forever the way people made sense of the world around them.

I saw them hanging together for the first time at the major Matisse retrospective that opened at the Museum of Modern Art in 1992.Their collective impact was even more startling when the exhibition crossed the Atlantic to the Beaubourg in Paris. I spent weeks on end looking at the show in both places. The cloth came and went on Matisse’s canvases, crashing in and out like a turbulent and disruptive theme in music. I didn’t know its story then, but I was fascinated by the way it seemed to beckon and drive the painter forward toward abstraction.

Matisse painted it first in Pierre with Bidouille in 1903, part of a picture of his 4-year-old son clutching a toy horse. The rough, jabbing blue marks on the cloth covering the table beside the child are barely recognizable as the conventional arabesques and flower baskets woven into the original fabric, but the stuff itself still looks like a relatively straightforward tablecloth. Other pictures show it lurking unobtrusively in the background as a curtain, or a screen.

But soon the “toile de Jouy” starts to stir and swell on Matisse’s canvases. In the first of two “Still Lifes With Blue Tablecloth” of 1905-06, its forking blue arabesques infiltrate the pots and bowls on the table, licking up the stem of a china fruit stand, outlining the rim of a water jug and collecting in a pool of color beneath a saucer, before finally shooting off the tabletop as a jagged blue streak in the top part of a canvas that no longer makes any serious pretense at representing a realistic interior. In a second variation on the same theme, the entire still life – coffee pot, fruit bowl, decanter – has been swallowed up by the fierce rhythmic patterns of a cloth that now looks more like a choppy heaving sea.

No one could mistake the electric energy in these pictures. The junk-shop material released a surge of tidal power that enabled Matisse to smash through the barrier that blocked his way forward as a painter. The same thing happened again and again with different textiles in these years, during which his canvases alarmed contemporaries as much as the parallel experiments of the Cubists.

In 1906 he was captivated by a batch of red, yellow, and black prayer mats picked up in Algeria. In 1908 it was a Persian rug, which he couldn’t initially afford (in the end his wife raised the money by taking a set of much-needed, brand new bed sheets back to the department store). Two years later it was a woven coverlet from the Sierra Nevada in Spain. All of them reappeared on canvases that replaced the old, worn-out conventions of perspective, tonal values, and three-dimensional illusion with a new and vigorous pictorial language.

It has been 13 years now since I had the germ of an idea for “Matisse, His Art and His Textiles,” a show that sets out to explore the mysterious power textiles held for the painter and trace the paths they took to liberate his imagination. Textiles surrounded Matisse from the cradle to the grave. He painted them all his life, and used them repeatedly to kick-start his imagination at key points in his career. But the professional curators to whom I tentatively suggested my idea dismissed it initially as laughable, or outrageous.

This was pretty much the reaction of the Paris art world before World War I to Matisse’s crazy painting. His canvases were mocked in London, jeered at in New York, and burned in effigy by art students in Chicago. The one person who responded with unreserved enthusiasm was the man who imported Matisse’s art to Russia, and turned his house into the world’s first museum of modern art, the great Muscovite collector Sergei Shchukin.

Shchukin had no problem deciphering pictures that looked to other people like a barbaric jumble of form and color. Ill-educated, uncultivated, treated by Parisian dealers as an ignorant boyar, Shchukin had no art-historical preconceptions. He was a textile manufacturer who had inherited a family firm and turned it into one of Russia’s leading industrial empires. He was familiar from infancy with the language of abstract pattern and Oriental decoration. So was Matisse, who had been born and brought up in the textile towns of northern France.

The two men developed a working partnership that was more like a collaboration than the usual commercial transaction between artist and collector. Shchukin’s organizing ability was phenomenal, and his excitement was contagious. He showed his collection personally, stammering in his eagerness, running ahead through the rooms, flinging open doors, making his bewildered visitors screw up their eyes to squint at Matisse’s colors, urging them to imagine the whirling curves of the “toile de Jouy” reaching out beyond the edges of the canvas into infinite space.

Nearly all the most radical paintings of this period belonged to Shchukin, whose collection was cut short by the outbreak of war in 1914, confiscated by Lenin in 1917, and locked away in cellars by Stalin in World War II, re-emerging only slowly in the late 1960s. Several of these paintings can now be seen for the first time in London at the RA exhibition.

The second revelation of this show is Matisse’s own textile collection. He called it his working library, and used it to turn his studio in the 1920s and 1930s into a treasure house of cushions, carpets, coverlets, African hangings, Arab embroideries, and costumes for the models. These have spent half a century since the painter’s death shut up in family chests and dressing-up boxes.

Now they have been reunited with the canvases they inspired in a context that puts a different spin on the generally dismissive view taken of Matisse’s “odalisque” period until now.

People have traditionally underrated Matisse’s paintings of models in see-through blouses and flimsy harem pants, lounging against cushions on striped divans or armchairs. He himself saw them as a particularly intense phase of his passionate, lifelong affair with color. “Matisse, His Art and Textiles” makes it possible to look again at these complex chromatic experiments, designed to help Matisse think in color as fast and fluently as he used line in drawing.

The exhibition ends with a final unexpected transition, when a group of woven grass-cloths from the Congo pointed the way towards the brilliant cut-paper compositions of Matisse’s last years. Once again, textiles acted on him as a liberating force. He said he had at last recaptured a dream of light and color first glimpsed in the dark smoky textile towns of his northern boyhood. “Even if I could have done, when I was young, what I am doing now – and it is what I dreamed of then – I wouldn’t have dared.”

“Matisse, His Art and His Textiles: The Fabric of Dreams” at the Metropolitan Museum of Art from June 24 until October 17 (1000 Fifth Avenue, at 82nd Street, 212-535-7710). Hilary Spurling’s “Matisse the Master” (Alfred A. Knopf) will be published this September.