Dilemma of hate

For those not aware, yesterday I had a run in with a guy selling nazi merchandise, this was not a piece of militaria, was not historic, it was a red banner with a white circle and swastika along with a picture of Hitler printed on the red with the dates he was in power.
The only reason for this to exist was for modern NAZI’s to celebrate Hitler, and to promote fascist beliefs.
I asked for it to be removed as the car boot is multicultural with people from all cultures including Jewish, Roma, Muslim, Christian…. The list goes on, there were also many disabled people their too, remember nazi eugenics? My request to the stall holder was refused, I was persistent and admit I ended up raising my voice to him as he defended the banner. A woman told me I was more offensive than Hitler and another stall holder threatened to take me down the field and beat me up.
None of the other customers at the sale defended my stance even though they stood around watching the scene. My friend however did
My best friend and carer was with me and she too felt threatened by the mob mentality.
The car boot organiser finally but reluctantly asked the stall holder to take the banner down then belittled me for challenging him. He also went on to say people could sell and display anything they wanted to. I explained that I had been threatened by another stall holder too and again was belittled by the organiser a man on his fifties who at this point reverted to a 15yr old school ground bully.
I had called the police as felt we were about to be assaulted but as usual they failed to attend.
My post about the events has gone viral and I have had many positive responses to my actions, however I also have had a very few negative ones, not I hasten to add by anyone that knows me.
I did not challenge this hate for fame, nor for any other motive than refusing to accept the rise of fascism in Britain and this modern world.
I have been anti fascist all my life and was active in the 1980’s with Anti-Nazi League and other anti hate groups.
I have worked all over the UK with all cultures, faiths, abilities and have always been anti discriminatory in my practice and my life.
I am not a terrorist though for some reason Antifa is seen as a terrorist organisation yet seemingly many fascist ones are not.
My dilemma is that now I have made a stand people want to stand with me, alongside me and want me to be interviewed, be public in regard to my actions. I am being warned against this by the people that care about me, as the Right Wing are often violent, aggressive and threatening. That by making my stand and continuing to stand against hate I am not just putting my self at risk, a risk I am willing to take, but also putting my family and friends in harms way.
As I said I am not scared for me, but after losing my young brother I cannot ever consider losing anyone else, especially through my actions and beliefs. My stubborness.
So do I carry on pushing to change our society at the very least, which I will always so in some way, or do I put my head down and live ashamed of what I might have done.
Yes backing down may also help my clinical anxiety which currently  is extreme due to yesterday, but not my depression nor my self respect.
So what the hell do I do. I did not plan to challenge this, but there was no reality where I could have just walked by.
I have always been proud of my grandparents, living through and fighting in the second world War. Whatever the government reasons for war were, the British people fought and died alongside of all the other allied nations to rid the world of the blind hate fascism brings and pushes upon us.
If I don’t now make my stand how on earth can I hold my head up to their memory or that of Rob my brother, who for all his faults shared my belief in fighting hate.
Jonesy