The thing about St. George's Day is that it can't have the same significance that St. David's Day or St. Andrew's Day or, even, St. Patrick's Day have. Culturally speaking, England is the dominant member of Great Britain. England is where the majority of British people live, where the main hub of government is based, where the Royal family live, where the most commonly spoken language comes from and where the majority of British media is made, filmed, based and broadcast from, etc.
Because of this, the national days of Scotland and Wales (and even the Rep. of Ireland too, even though they obviously aren't British), take on more significance than St. George's Day because they provide a chance for people to celebrate non-hegemonic (English) culture. In England, we don't really need a day to celebrate being 'English', when 'Englishness' is the default for everyday life in Britain, generally speaking.
Then of course, there's the issue of far-right, racist, xenophobic cunts abusing the concept of 'English nationalism' to push their hateful agendas about who should or shouldn't be accepted in our society. Which, understandably, puts a lot of people off.