Ferrets at dinner time??

my ferret has a three story condo in the dinning room. how this is the only safe place to put his cage couse of my silly puppy allways bother him other wise. now i let him out several times a day to play and roam free while dogs are other wise detained in the other room so he isn't tormented. he is out about 3 4 hours a day. that said, every night at dinner time, my dinner time, when we all sit down to eat he goes nuts! pulling at the bars and makeing a ton of noise. i tryed keepng him out while i made dinner which takes about an hour every night, then puting him away when we sit down to eat, with fresh food for him in his bowl. but it doesn't help he still goes bonkers. what can i do to stop this behavior?? it's ruining our dinners.