Most fascinating though, are the women-only mosques, and even more surprising is that they have female prayer leaders - women imams.
The main women's mosque is close to the central men's mosque, across an alley lined with food stalls with steaming tureens and white-capped bakers making the local spiced bread.
The prayer leader here is Guo Jingfang, who was trained by her father, an imam at the men's mosque.
She took me through Kaifeng's winding alleys, stopping on the way to hold animated chats with neighbours and to pick up an order from the local cake maker, until finally we came to the ornamental gate of what looked like a little Confucian temple. Inside was a tiny flagged courtyard with a tiled roof festooned with vines and yellow flowers.
This is Wangjia Alley mosque, said to be the oldest surviving women's mosque in Kaifeng, built in 1820. The prayer hall is scarcely more than a spacious living room covered with carpets and chairs. It could hardly fit more than 50 people but it is one of the loveliest places of worship I have seen anywhere.
Outside, in dappled sunlight, we met members of the community and their prayer leader. Once a factory worker, she came from a religious family and after five years of study had become an ahong - a woman prayer leader - though she sees her main job simply as teaching women to read the Koran.
We stood in the courtyard and chatted away. Guo Jingfang saw women's mosques as a Chinese tradition but especially strong in Henan - there are 16 in Kaifeng and dozens more in the countryside around, along with small teaching schools in the big city, Zhengzhou, and in some smaller towns. Further afield, there are many more down south in Yunnan and in the north, but not in Muslim Xinjiang, where they follow a more traditional Central Asian brand of Sunni Islam.
As for how the tradition of women's mosques started, we have to go back to the founding of the Ming Dynasty in the late 1300s, when the Muslim community - previously favoured guests - suddenly became an anxious and oppressed minority. Responding to the shock of the alien Mongol occupation, the early Ming rulers waged a chauvinistic war against non-Han peoples. Minorities now aroused hostility and suspicion and were subject to a brutal policy of assimilation - the Muslims were told they must marry Han people and not among themselves.
So the 15th Century was almost catastrophic for Chinese Islam. But in the late 16th Century things improved and among the Muslims a new cultural movement began, a revival of Islamic culture and education.
A century later Chinese Muslim philosophers were able to write erudite books showing how you could be a loyal Muslim and also loyal to the Chinese state. And at this point, at the grassroots, men realised how important women could be in preserving and transmitting the faith.
So women's mosques grew out of a double movement in the Chinese Muslim world - the need to preserve the community, and the desire for women's education.
Guo Jingfang and her friends in Kaifeng think that the schools came first, and then became full mosques in the 18th Century. Education still has a big role today, from basic teaching to copying texts.
"When our mothers were girls it was the only place where poor Muslim women could receive an education: the women did it together, women supporting women," said one of the women chatting in the mosque's courtyard.
"In some places in the Muslim world it is not allowed, but here we think it a good thing. Women have had a better status here since 1949 and this is part of it."
One of the women mentioned the progressive ideas of the Islamic Association of Kaifeng, which gets men and women to work together on new education projects.
"China is changing and these are good things for the future," she said.
Later, in the main women's mosque, everyone joined in the prayers, and the men in our crew were invited too, visitors from afar.
There were 30 or so women, young and old, in coloured and embroidered headscarves, lime-green, scarlet, black spangled with silver stars. After a period of calm reflection, Guo Jingfang lifted her hands and began singing. Then facing the congregation she started the prayers. It was beautiful and simple, the sounds of the street receding so you could almost hear a pin drop. I felt privileged to be there.
Afterwards, everyone crowded round. Half our crew were women - director Rebecca, translator Qian, and assistant producer Tina Sijiao - and Guo Jingfang and her friends were as fascinated about their lives as we were about theirs. We ended with laughter and selfies - jolly pictures of pious, thoughtful, joyful women, comfortable in their skins and strong in their solidarity for women.
The Muslim community has had its ups and downs in China. Today, as in the Ming Dynasty, they are anxious to demonstrate their loyalty, as rumours come of dissent in the far west in Xinjiang - keen to point out that the old Muslims, the Hui, are loyal Chinese, just as the Han are.
But what began as a response to the patriarchy of traditional Chinese society, and to the historical situation of Islam under the Ming and the Manchus, has now also become a feminist issue.
China was isolated for much of the 20th Century, so these women-only mosques were untouched by the waves of radical Islam which came after the 1979 Iranian Revolution. Now the wheel is turning again and what is seen as completely normal here in Kaifeng is being taken up in other parts of the world. Iran accepted women's mosques not so long ago. There are women's mosques in Berlin and Amsterdam, in Lebanon and Bukhara and even in Sudan. An African-American woman recently led prayers in the USA's first-ever women's mosque in Los Angeles. In the UK, the Bradford Muslim Women's Council only recently announced a plan to build the UK's first women's mosque.
To many Muslims I have spoken to, men and women, the movement is an inevitable necessary and renewing phase in the history of Islam… And if the goal is the renewal of spiritual life then the gorgeous little mosque in Wangjia Alley carries a lesson for us all.
Travelling writing and filming for the last 35 years between the Mediterranean and the Yellow Sea, I have often had occasion to reflect on how frequently the Muslim world is misunderstood, in its beliefs, but also in its customs and practices.
It's amazing isn't it how often an expert claims to talk about Islam as if it were monolithic? Sometimes you might even think the Wahhabis spoke for the majority, when actually they are a minority in a vast sea of faith with myriad points of view.
Think of the beautiful shrines of the Punjab with their haunting qawwali music. Or the other side of the coin, the old moulids - the exuberant saints' fairs of Egypt with their all-night dancing, ferris wheels and medicine shows… the spiritual and the worldly side by side, taking energy from each other.
On my travels over the years - to the Sufis in Konya (Turkey) or Sind (Pakistan), shrines like Nizamuddin in Delhi, Chishti in Ajmer (Rajasthan), or Ibn al-Arabi in Damascus, the timeless calm of Al-Azhar (Cairo), the passion of Karbala (Iraq) - I have always been struck by how full of rich difference the world of Islam is. And among those unforgettable experiences the women's mosques in Kaifeng will have a very special place.