These words you should know when communicating with Russian people. This expression means: I am not sure, but most probably no.
Or: I am not sure, but I think no.
People can reply to you like that in many cases. For example if you ask:
"Вы будете чай?" - "Would you like some tea?"
"Да нет, наверное." - "I think no."
But if they answer so, it means they are not sure in their answer. It is not a firm No. So if you really want them to do that thing, you still can persuade them. Just say:
"Ну давай! Немножко! За компанию." - "Oh common! Just a bit! For company."
Or something like this which will suit your situation.