Totally depends what you end up working on as a programmer. If it's web apps, you'll be totally fine. All you need is basic arithmetic. Writing a game engine? You'll need to know some basic to moderate matrix maths...
If you're doing formal verification using unbounded model checking... good fucking luck.
On average I would say most programming tasks need very little maths. If you can add and multiply you'll be fine. Definitely sounds like you'll be ok.