The Manchester City star is firmly established as Roy Hodgson's first choice but Banks is concerned there is no one currently pushing him for his place at either club or international level.
Romanian Costel Pantilimon and veteran Richard Wright provide the back-up at City, while Hodgson has also selected the inexperienced John Ruddy from Norwich and Celtic's uncapped goalkeeper Fraser Forster for the forthcoming FIFA World Cup qualifiers against San Marino and Poland.
"Joe is the best English goalkeeper by a mile, no question, but what concerns me is that there is nobody challenging him for that position," said Banks, who won 73 England caps.
"I don't think he is the best in the world but the age he is (25), he stands a great chance of being the best. I don't want him to get too over-confident where he doesn't have to worry about his place. Nobody is pushing Joe Hart at all - he is not even being pushed by his club.
"Peter Shilton and Ray Clemence used to share the duties and they had Phil Parkes, Joe Corrigan and Peter Bonetti (challenging) and when I came through there was Ron Springett.
"You couldn't have a bad game and then say to yourself that you'd have to be in the squad because there is nobody else, you had to be on your toes.
"We currently have no strength in depth, none whatsoever and the reason is that with the Premier League you have foreign players taking the position of young home-grown lads.
"There are fewer and fewer players to pick from. We have got to find a method of bringing young goalkeepers through.
"If you are a goalkeeper your standards can't be raised (by playing alongside foreigners) because you have to be in the side to play. You are either in the team or you are not."
Room for further improvement
Banks believes Hart still has some areas of his game which require work but is confident he can iron out his flaws and follow in the footsteps of Real Madrid and Spain goalkeeper Iker Casillas, considered by many to be the world number one.
"He is still only a young man and he can still learn a lot and when I see him there is a lot to admire: his positioning and the way he is prepared to come off his line are excellent," added Banks.
"Okay, he might misjudge one or two crosses but that will come with experience because for a big man he is very agile.
"Crosses are the only weakness to his game but it is more difficult for a goalkeeper now because of the lightness of the ball and the way it moves in the air.
"I used to pride myself on my positional play and Hart's is good but positioning sometimes does not make the slightest bit of difference when you see that ball start to swerve. It makes it very hard to judge if you are going to catch it or punch it away."
Banks said he thought part of the reason there is a lack of talent coming through is due to the fact youngsters now do not see being a goalkeeper as an attractive proposition.
"I think the position has lost its allure," said the 1966 FIFA World Cup winner, speaking at the launch of the 'In the 90th' campaign, marking the 90th season of the Football Pools.
"You imagine you are a young kid facing a ball that swerves and dips. You'd get discouraged pretty quickly. You'd soon want to be the one doing the shooting."